I have a problem with infinite number of solutions to maximal likelihood estimation. An example could be found from here Example 8.3. I will rewrite this example in the following,
Suppose that $Y \sim Normal(X\theta,I)$, where $X$ is an $n\times k$ matrix with rank smaller than $k$ and $\theta \in \Theta \in R^k$. The density function is
$$p(y;\theta)=(2\pi)^{-n/2}exp(-\frac{1}{2}(y-X\theta)'(y-X\theta))$$
Since $X$ is not full rank, there exists an infinite number of solutions to $X\theta=0$. That means that there exists an infinite number of $\theta$'s that generate the same density function. So, $\theta$ is not identified. Furthermore, note that the likelihood is maximized at all values of $\theta$ satisfying $X'X\theta = X'y$.
I am curious in three questions. Firstly, are we still able to say the aysmptotic distribution of $\hat\theta_{MLE}$ is normal when $n \rightarrow \infty$. Secondly, whether there is a way to estimate the variance of $\hat\theta_{MLE}$? I want to use Fisher Information to do so, but it seems the FIM will also be singular. Thirdly, for the singular FIM matrix, I found this paper using Moore-Penrose Generalized Inverse to find CRB. It's a little beyond my knowledge so I want to make sure if my understanding is correct.
I can see the setting of this example is similar to $n<p$ problem in regression analysis, and I know people use regularization method to shrink some $\theta$'s to zero. But I am more curious if we don't use regularization and keep all the $\theta$'s, is it possible to estimate variance covariance between the $\theta$'s using Moore-Penrose Generalized Inverse of FIM?
Any explanations are appreciated!