I want to ask how to show that Linear Regression is linear invariant? The problem is specified in the following picture:
Here is the "solution" for the problem. But I really get confused by its second step: Why $(Z^TZ)^{-1} = Z^{-1}Z^{-T}$? The matrix Z is not a square matrix. I am wondering how this is legit? I will also greatly appreciate if anyone can give alternative solution to this problem. Thanks!
You're right: distributing the inverse as they did only works if $Z$ is square, which it is usually not.
A valid solution is as follows: \begin{align} \theta' &= (Z^TZ)^{-1}Z^Ty \\ &= ((XA^T)^T(XA^T))^{-1}(XA^T)^Ty \\ & = (AX^TXA^T)^{-1}AX^Ty \\ & = [A^{-T}(X^TX)^{-1}A^{-1}]AX^Ty \\ &= A^{-T}(X^TX)^{-1}X^Ty \\ &= A^{-T}\theta \end{align} The trick here is not to break up the product $X^TX$ when distributing the inverse.