what are the special cases of the covariance matrices () for EM algorithm for a Gaussian mixture

134 Views Asked by At

Consider 5 various forms of the covariance matrices in a Gaussian mixture model. The original problem was: Derive the EM equations for maximizing the likelihood function under such a model.

$$ p(x|\mu, \Sigma)=\frac{1}{(2\pi)^ \frac{d}{2}|\Sigma_k|^ \frac{1}{2} } exp[-\frac{1}{2} (x-\mu_k)^T \Sigma^{-1} (x-\mu_k)] $$

I know that to derive the EM equations, one should first set the covariance with respect to its form (given in the following), then compute the MLE. i.e. first compute the log-likelihood, second derivate w.r.t $\mu$ or $\Sigma$ or, $\alpha$. third set the derivative to zero and extract the desired variable.

$$ L(\theta)= \sum_{i=1}^n \sum_{k=1}^k \gamma(z_{ik}) [ \alpha_{k}+\frac{1}{(2\pi)^ \frac{d}{2}|\Sigma_k|^ \frac{1}{2} } exp[-\frac{1}{2} (x-\mu_k)^T \Sigma^{-1} (x-\mu_k)] ] $$

5 different types of covariance matrix

  1. Σk=Σ.
    this means that all of the elements in the matrix are equal. we should substitute the Σ with Σ to solve this case. \begin{equation*} \begin{bmatrix} a & a & a \\ a & a & a \\ a & a & a \end{bmatrix} \qquad \end{equation*}
  2. Σk=arbitrary.
    this means that the elements in the matrix have different values. we should substitute the Σ with Σk to solve this case. \begin{equation*} \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & i \end{bmatrix} \qquad \end{equation*}
  3. Σkj2I.
    this means that only diagonal elements are non-zero but they have different values. we should substitute the Σ with σd to solve this case.

    \begin{equation*} \begin{bmatrix} a & 0 & 0 \\ 0 & b & 0 \\ 0 & 0 & c \end{bmatrix} \qquad \end{equation*}

  4. Σk2I.
    this means that only diagonal elements are non-zero and they are all equal. \begin{equation*} \begin{bmatrix} a & 0 & 0 \\ 0 & a & 0 \\ 0 & 0 & a \end{bmatrix} \qquad \end{equation*}
  5. Σk=diag({σ2jk }dk=1 )
    I don't understand what form does it has? if it is diagonal, its diagonal elements are either equal (form 4), or they are different (form 3).what should I put instead of Σ to solve this case? \begin{equation*} \begin{bmatrix} ? & 0 & 0 \\ 0 & ? & 0 \\ 0 & 0 & ? \end{bmatrix} \qquad \end{equation*}

I understand cases 1 to 4. But, I have a hard time understanding case 5. any ideas? please draw a matrix and show the difference between the elements in case 5. and what I should put instead of Σ to solve this case?