Existence of maximum likelihood estimator in factor analysis

38 Views Asked by At

Let $Y_1,\dots,Y_n$ be i.i.d. $N(0,\Sigma$) random variables, where $\Sigma=FF'+ D$ with $F$ $m\times k$ and $D$ is diagonal positive definite. Let $V=\frac{1}{n}\sum_{i=1}^n Y_iY_i^\top $ and define the log-likelihood

$$vec(F)\times diag(D)\mapsto L(F,D)=-\frac{n}{2}\log|\Sigma|-\frac{n}{2}Tr(\Sigma^{-1}V)$$

on the parameter space $\Theta= \mathbb R^{mk}\times(0,\infty)^m$. Note that $\Sigma$ is positive definite and $L$ is continuous on the open set $\Theta$. In fact from the Woodbury formula we have $\Sigma^{-1}=D^{-1}-D^{-1}F(I_k+\Gamma)^{-1}F'D^{-1}$ where $\Gamma:=F'D^{-1}F$.

Using the rules of matrix differentiation one can compute

$$\frac{\partial L}{\partial diag(D)}=-\frac{n}{2}diag(\Sigma^{-1})+\frac{n}{2}diag(\Sigma^{-1}V\Sigma^{-1})$$ $$\frac{\partial L}{\partial F}=-n\Sigma^{-1}F+n \Sigma^{-1}V\Sigma^{-1}F$$

which shows that $L$ is continuously differentiable. Hence neccessary conditions for a local maximum are $\frac{\partial L}{\partial diag(D)}=0$ and $\frac{\partial L}{\partial F}=0$. Using the above two formulas one can show by algebra that these two conditions are equivalent to the following ones :

$$(1) \quad diag(FF'+D)=diag(V)$$

$$(2)\quad VD^{-1}F=F[\Gamma+I_k]$$

(See chapter 14 in the book An Introduction to Multivariate Statistical Analysis (2003) by Anderson). Note that if $Q$ is a $k\times k$ orthogonal matrix then (1) and (2) are still satisfied if we replace $F$ by $FQ$, i.e. the two conditions only determine $F$ up to a rotation. By choosing $Q$ to be an orthogonal matrix of eigenvectors for $\Gamma$ we can WLOG add the restriction that $\Gamma$ is diagonal to conditions (1) and (2).

Question: Are conditions $(1)$,$(2)$, and the condition that $\Gamma$ is diagonal, also sufficient for a local maximum?