I understand that, many classical methods for multiple regression won't work when $N<p$, where $p$ is the dimension of the input space and $N$ is the sample size.
For example, LSE for multiple regression, if $\mathbf{X}^T\mathbf{X}$ is nonsingular, then the unique solution is given by $\hat\beta=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$. But if $p>N$, $\mathbf{X}_{N\times p}$ cannot be of full column rank, thus $\mathbf{X}^T\mathbf{X}$ is singular and the parameters in the regression model cannot be uniquely estimated.
Now, my confusion is, Principal Component Regression is also a popular classical regression method for high-dimensional data, but will there be any problem, like above, when we apply PCR to a high-dimensional ($n\ll p$) data?
More specifically, if $\mathbf{X}^T\mathbf{X}$ is singular, how to calculate the principal components? (How to make the eigen decomposition for a singular matrix?) Are the principal components given by softwares still correct? Can we still regress on these principal components for dimension reduction purpose?
Could anybody give me a hint? Thanks a lot!
What I'm alsways doing is to do a "robust" cholesky-decomposition and then an iterative Jacobi-rotation to the principal components. "Robust" means here, that missing/low rank is handled in the same sense as zero-eigenvalues in the "pseudo-inverse" computation. So that "robust" cholesky decomposition gives a reduced set of components, which can then be made orthogonal by a column-wise rotation.
With a rank-3 random correlationmatrix r $$ \text{ R =} \small \left[ \begin{array} {rrrrrr} 1.0000& -0.8790& -0.8389& -0.8184& -0.5239& -0.7192\\ -0.8790& 1.0000& 0.6751& 0.5508& 0.0846& 0.3115\\ -0.8389& 0.6751& 1.0000& 0.9721& 0.7127& 0.7833\\ -0.8184& 0.5508& 0.9721& 1.0000& 0.8534& 0.9053\\ -0.5239& 0.0846& 0.7127& 0.8534& 1.0000& 0.9633\\ -0.7192& 0.3115& 0.7833& 0.9053& 0.9633& 1.0000 \end{array} \right] $$ I get by the "robust" cholesky-decomposition the loadings-matrix $L$ with only 3 significant columns $ L = cholesky(R)$ $$ \text{ L =} \small \left[ \begin{array} {rrrrrr} 1.0000& 0.0000& 0.0000& 0.0000& 0.0000& 0.0000\\ -0.8790& 0.4767& 0.0000& 0.0000& 0.0000& 0.0000\\ -0.8389& -0.1308& 0.5283& 0.0000& 0.0000& 0.0000\\ -0.8184& -0.3536& 0.4530& -0.0000& 0.0000& 0.0000\\ -0.5239& -0.7886& 0.3218& 0.0000& -0.0000& 0.0000\\ -0.7192& -0.6727& 0.1742& -0.0000& -0.0000& 0.0000 \end{array} \right] $$ and by iterative columnwise jacobi-rotation on the first 3 nonzero axes $ P=\operatorname{rot}(L,\text{"PCA"})$ the principal components-solution $$ \text{ P =} \small \left[ \begin{array} {rrr} 0.9037& -0.3761& 0.2045\\ -0.6478& 0.7610& -0.0363\\ -0.9557& 0.0870& 0.2812\\ -0.9832& -0.1061& 0.1482\\ -0.8119& -0.5821& -0.0446\\ -0.9087& -0.3731& -0.1871 \end{array} \right]$$ At the moment I do not exactly how to proceed to the regression-step - perhaps please give a hint how your model of dependent/independent variables is thought to be organized.