Question about projection onto subspaces

Question

Suppose I have an $n\times n$ (complex) matrix $Q$ viewed as a linear transformation from $\mathbb{C}^{n}$ to $\mathbb{C}^{n}$. I'm struggling to understand the difference between the following two scenarios. For simplicity, let us consider $n = 4$.

Scenario 1: Let $S$ be a subspace of $\mathbb{C}^{4}$ of dimension $d=2$. Let us think of $S$ as the subspace generated by two eigenvectors $v_{1}$ and $v_{2}$ of $Q$.One can construct a projection matrix $P_{S}$. According to this post, this projection should be given by: $$P_{S} = A(A^{T}A)^{-1}A^{T}$$ where $A = [v_{1} \hspace{0.1cm} v_{2}]$ the a $4\times 2$ matrix. This is a $4\times 4$ matrix.

Scenario 2: Let us write $\mathbb{C}^{4} = S \oplus S^{\perp}$. Since $Q$ is a linear operator on $\mathbb{C}^{4}$, it might be possible to decompose $Q = Q_{1}\oplus Q_{2}$ so that, if $u = u_{1}+u_{2}$ and $u_{1}\in S$ and $u_{2} \in S^{\perp}$: $$Qu = Q_{1}u_{1}\oplus Q_{2}u_{2}$$ Each $Q_{i}$ should be a $2\times 2$ matrix.

Here is my problem. I would expect that $Q_{1} = QP_{S}$, since $Q_{1}$ should be the restriction of $Q$ to $S$. However, as I stressed before, both $Q$ and $P_{S}$ are $4\times 4$ matrices. So, where is the gap? If $Q_{1}$ is not $QP_{S}$, how can I obtain $Q_{1}$? And finally, why is $P_{S}$ a $4\times 4$ matrix if it projects into a subspace of dimension $2$? (I would expect it to be a $2\times 4$ matrix instead).

Answer 1

If $V$ is a finite dimensional inner product space (like $\mathbb{C}^{n}$) and $S$ is a subspace, we have two similar notions. There is the orthogonal projection operator $P_{S}:V\rightarrow V$ that intuitively takes $x\in V$ to the nearest element of the subspace $S$, but it is still a mapping $V\rightarrow V$. This operator satisfies $P^{2} = P$, $P = P^{*}$, and $\text{Im}(P)=S\subset V$, and it happens that $I-P_{S} = P_{S^{\perp}}$. There is also the decomposition $V = S\oplus S^{\perp}$ and the mappings $\pi_{S}:V\rightarrow S$ and $\pi_{S^{\perp}}:V\rightarrow S^{\perp}$ that map $v\in V$ to $P_{S}(v)\in S$ or $P_{S^{\perp}}v\in S^{\perp}$ respectively. Let's suppose that $V$ has dimension $n$ and $S$ has dimension $k$. As functions, $P_{S}$ and $\pi_{S}$ map $x$ to the same element, but they have different codomains $V$ and $S$ respectively. When we write $P_{S}$ and $\pi_{S}$ as matrices, they are $n\times n$ and $k\times n$ respectively.

In a moment, we will need the inclusion mappings $i_{S}:S\rightarrow V$ and $i_{S^{\perp}}:S^{\perp}\rightarrow V$ that simply map a vector in the subspace to the same vector in the larger vector space $V$. It may be helpful to observe that even though these mappings essentially "do nothing", they are represented by rectangular (nonsquare) matrices.

Given a linear operator $Q$, we cannot generally decompose it as $Qu = Q_{1}u\oplus Q_{2}u$ like you wrote in your post (unless you interpret the notation $\oplus$ strangely). We can decompose it into a block matrix with four blocks. We should choose a basis $V$ of the form $s_{1},\ldots,s_{k},q_{1},\ldots,q_{n-k}$ so that $s_{1},\ldots,s_{k}$ is a basis of $S$ and $q_{1},\ldots,q_{n-k}$ is a basis of $S^{\perp}$. Then

$$Q = \begin{pmatrix}\pi_{S}Qi_{S} & \pi_{S}Qi_{S^{\perp}}\\ \pi_{S^{\perp}}Qi_{S} & \pi_{S^{\perp}}Qi_{S^{\perp}}\end{pmatrix}.$$

If $V$ is $n$-dimensional and $S$ is $k$ dimensional, then we have cut up the $n\times n$ matrix $Q$ into (clockwise) an $k\times k$, $k\times (n-k), (n-k)\times (n-k)$ and $(n-k)\times k$ submatrices.

We may also write it in a very similar way as a sum of four operators (square matrices mapping V to itself). Remember that $P_{S^{\perp}} = I - P_{S}$. Then $Q = IQI = (P_{S} + I-P_{S})Q(P_{S}+I-P_{S}) = P_{S}QP_{S} + P_{S}QP_{S^{\perp}} + P_{S^{\perp}}QP_{S} + P_{S^{\perp}}QP_{S^{\perp}}$.

To give an example to relate these concepts, observe that

$$P_{S}QP_{S^{\perp}} = \begin{pmatrix}0 & \pi_{S}Qi_{S^{\perp}}\\0 & 0\end{pmatrix}.$$

Finally, I will try to answer your problem concretely. You wrote "$Qu = Q_{1}u \oplus Q_{2}u$" where $\oplus$ refers to the decomposition $V = S\oplus S^{\perp}$. I think this is an improper use of this notation, since if we write $x = y\oplus z$, it should be that $y\in S$ and $z\in S^{\perp}$, but $Q_{1}$ and $Q_{2}$ will not necessarily produce values in $S$ and $S^{\perp}$ respectively. [If they do, then the decompositions above are block diagonal, which is an important special case.] The closest thing to $Q_{1}$ from your post would be $Qi_{S}$, which takes vectors from $S$ to $V$ and is, therefore, $4\times 2$ (a mapping from $\mathbb{C}^{2}\rightarrow \mathbb{C}^{4}$). Compare the block decompositions of $QP_{S}$ and $Qi_{S}$ below:

$$QP_{S} = \begin{pmatrix}\pi_{S}Qi_{S} & 0\\ \pi_{S^{\perp}}Qi_{S} & 0\end{pmatrix}\in \mathbb{C}^{n\times n},$$ and $$Qi_{S} = \begin{pmatrix}\pi_{S}Qi_{S}\\\pi_{S^{\perp}}Qi_{S}\end{pmatrix}\in \mathbb{C}^{k\times n}.$$

You also ask "why is $P$ a $4\times 4$ mapping if it projects to a subspace of dimension two?" Projections refer to $P_{S}$ not $\pi_{S}$. I think you are confusing these two mappings. They are essentially the same, except that they have different codomains. That makes them very different when you represent them as matrices.

Answer 2

I started working on a trivial example before the much better answer appeared, but perhaps this simple concrete example will help guide someone.

You can express tensors in different bases. Just like I can express the 2D vector at $v = 3 \mathbf{e}_x + 4 \mathbf{e}_y$ in terms of basis vectors for an $x-y$-plane, I can also express it in terms of $v = 5 \mathbf{e}_v$ where $\mathbf{e}_v = \frac{3}{5} \mathbf{e}_x + \frac{4}{5} \mathbf{e}_y$.

When we only pass around matrices with just the coefficients, e.g. $v = [3,4]$ there is an assumption made of what base they are expressed in. If we want to project another vector, e.g. $u = 5 \mathbf{e}_x + 0 \mathbf{e}_y$ we can express this projection in several ways, in terms of the 2 coefficients of $\mathbf{e}_x$ and $\mathbf{e}_y$, or in terms of the 1 coefficient of $\mathbf{e}_v$. Working through the computations for the case I made up here, one will get $$ u_v = \frac{9}{5} \mathbf{e}_x + \frac{12}{5} \mathbf{e}_y \equiv 3 \mathbf{e}_v $$ Either way to tackle it, it's the same vector, we only need to think of what base we are expressing the coefficients in.

Answer 3

There is something fiddly happening here which basically boils down to the fact that you can view $Q_1$ (and $Q_2$) as matrices in $GL(2;\mathbb{C})$ since they act on a $2$ dimensional space, or view them as elements of $GL(2;\mathbb{C}) \subset GL(4;\mathbb{C})$ i.e. view them as $4 \times 4$ matrices.

As $S$ is viewed as a subset of $\mathbb{C}^4$ it is a good idea to view $u_1$ and $u_2$ as vectors with four entries even though $u_1 \in S$ and $u_2 \in S^{\perp}$, which are both $2$ dimensional subspaces of our original space.

As in example, if our space is $\mathbb{C}^4$ and we consider $S = \{e_1,e_2\}$ where $e_1$ and $e_2$ are the standard basis vectors then we would still want to write, $e_1 = \begin{pmatrix} 1 \\0\\0\\0 \end{pmatrix}$ rather than $\begin{pmatrix} 1 \\ 0 \end{pmatrix}$ even though $e_1 \in S$, where $S$ is a $2$-dimensional space, due to the fact that we view $S$ as a subspace of $\mathbb{C}^4$.

Thus, we need $Q_1$ and $Q_2$ to be $4 \times 4$ matrices, it might look something like this:

$Q = \begin{pmatrix} 1&0&0&0\\0&4&0&0\\0&0&5&6\\0&0&7&8 \end{pmatrix} = \begin{pmatrix} 1&0&0&0\\0&4&0&0\\0&0&0&0\\0&0&0&0 \end{pmatrix} + \begin{pmatrix} 0&0&0&0\\0&0&0&0 \\0&0&5&6\\0&0&7&8 \end{pmatrix} = Q_1 + Q_2$.

So yes, if you view $Q_1$ as a $4 \times 4$ matrix then you will have $Q_1 = QP_S$ (and someone please correct me if I'm wrong, but I don't see why $Q_1 = P_SQ$ also wouldn't work).