Show that if $||Y - X_j\hat{\beta}_j||^2_2 < ||Y - X_{j^*}\hat{\beta}_{j*}||^2_2$ then $X_j^TY/||X_j||_2 \geq \pm X_{j*}^TY/||X_{j*}||_2$

84 Views Asked by At

Suppose $Y \in \mathbb{R}^n$, $X \in \mathbb{R}^{n \times p}$, $\beta \in \mathbb{R}^p$ and $\epsilon \in \mathbb{R}^n$ with mean 0 and covariance $\sigma^2I_n$. Assume the following relationship between $X$ and $Y$: $$Y = X\beta + \epsilon$$

Suppose that $p=3$, and $X_j \in \mathbb{R}^n$ is the $j$th column of the matrix $X$ for $j = 1, 2, 3$. Let $\hat{\beta}_j = \left(X_j^TX_j\right)^{-1}X_j^TY$ be the least squares estimator.

I want to verify the following statement:

"If $\hat{\beta}_3$ has the lowest residual sum of error, i.e., $$||Y - X_3\hat{\beta}_3||^2_2 < ||Y - X_2\hat{\beta}_2||^2_2 \text{ and } ||Y - X_3\hat{\beta}_3||^2_2 < ||Y - X_1\hat{\beta}_1||^2_2,$$ then $$s_3X_3^TY/||X_3||_2 \geq \pm X_1^TY/||X_1||_2 \text{ and } s_3X_3^TY/||X_3||_2 \geq \pm X_2^TY/||X_2||_2,"$$ where $s_3 = sign(X_3^TY)$.

My thought process is as follows: Since $X_j$ is a vector, then $\hat{\beta}_j = \left(X_j^TX_j\right)^{-1}X_j^TY = X_j^TY/||X_j||^2_2$. However, I'm not sure where to go from here.

1

There are 1 best solutions below

0
On BEST ANSWER

Let $Z_j:=X_j^{\top}Y/\|X_j\|_2$. Then $$ \|Y-X_j\hat{\beta}_j\|_2^2=Y^{\top}(I-X_j(X_j^{\top}X_j)^{-1}X_j^{\top})Y=\|Y\|_2^2-Z_j^2, $$ and $$ \|Y-X_3\hat{\beta}_3\|_2^2\le \|Y-X_j\hat{\beta}_j\|_2^2 $$ is equivalent to $$ Z_3^2\ge Z_j^2 \quad\Leftrightarrow\quad |Z_3|\ge |Z_j|. $$