Relationship between projection of $y$ onto $x_1, x_2$ individually vs. projection on both?

435 Views Asked by At

This is essentially similar to the question I just asked on cross validated, but here I am going to pose it in a linear algebra way.

Consider $y \in \mathbb{R}^n$ and $x_1, x_2, 1_n \in \mathbb{R}^{n}$. Suppose you orthogonally project $y$ onto $x_1, 1_n$ and find the projection of $y$ onto the subspace spanned by $x_1, 1_n$ can be written as $\hat{y}_1 = \hat{\beta}_1 x_1 + b_1$, i.e., a linear combination of $x_1$ plus some offset. Now do the same for orthogonal projection of $y$ onto $x_2, 1_n$ and find $\hat{y}_2 = \hat{\beta}_2 x_2 + b_2$.

Now consider projecting $y$ onto the subspace spanned by both $x_1, x_2, 1_n$ and find $\hat{y}_{12} = \hat{\gamma}_1 x_1 + \hat{\gamma}_2x_2 + b_{12}$.

If $x_1 \perp x_2$, then I know $\hat{\beta}_i = \hat{\gamma}_i$. But what if they're not orthogonal?

What can I say about the relationship between $\hat{\beta}$ and $\hat{\gamma}$ in this case?

Some specific questions that I am also interested in is if $\hat{\beta} >0 $, does this imply $\hat{\gamma} > 0$? If $x_1, x_2$ are linearly dependent, then I don't think this won't be true for one of the coefficients.

1

There are 1 best solutions below

1
On

I cannot say I completely understood what those constants $b_1$, $b_2$ or $b_{12}$ are for. But I understood the gist of your question and I'll try my best.

Say the orthogonal projection of $y$ onto the subspace spanned by $x_1$ can be written as $\hat{y}_1 = \hat{\beta}_1 x_1$, i.e., a linear combination of $x_1$. Now we do the same for orthogonal projection of $y$ onto $x_2$ and find $\hat{y}_2 = \hat{\beta}_2 x_2$.

Also we have the projection of $y$ onto the subspace spanned by both $x_1, x_2$ and find $\hat{y}_{12} = \hat{\gamma}_1 x_1 + \hat{\gamma}_2x_2$.

Without loss of generality we can say the vectors $x_1$ and $x_2$ are unit vectors and represent them by $\hat{x_1}$ and $\hat{x_2}$. If you don't want to do this, then rewrite all the vectors in terms in terms of $\hat{x_1}$ and $\hat{x_2}$. So for example, $\hat{\beta_1}$ will become $\hat{\beta_1} ||x_1||$

Now, consider this statement. The orthogonal projection of $\hat{y_{12}}$ onto $x_1$ would be the same as $\hat{y_1}$ and the orthogonal projection of $\hat{y_{12}}$ onto $x_2$ would be the same as $\hat{y_2}$.

So, by the definition of projection,

$$ \hat{y_{12}}.\hat{x_1} = ||\hat{y_1}|| $$

$$ \implies (\hat{\gamma}_1 \hat{x_1} + \hat{\gamma}_2 \hat{x_2}).\hat{x_1} = ||\hat{y_1}||$$

$$ \implies \hat{\gamma}_1 \hat{x_1}.\hat{x_1} + \hat{\gamma}_2 \hat{x_2}.\hat{x_1} = \hat{\beta}_1$$

$$ \implies \hat{\gamma}_1 + \hat{\gamma}_2 \hat{x_2}.\hat{x_1} = \hat{\beta}_1 \tag{1}$$

Similarly we can solve $ \hat{y_{12}}.\hat{x_1} = ||\hat{y_2}|| $ to get

$$ \implies \hat{\gamma}_1 \hat{x_1}.\hat{x_2} + \hat{\gamma}_2 = \hat{\beta}_2 \tag{2}$$

There you go. We have 2 equations and 2 unknowns.

Obviously we should know the value of $\hat{x_1}.\hat{x_2}$, in other words the cosine of the angle between them, to get the required relations. In the case where $\hat{x_1}$ and $\hat{x_2}$ are orthogonal, $cos \frac{\pi}{2}=0$ and hence the result you gave $\hat{\beta}_i = \hat{\gamma}_i$.