I need help solving this following equation, a Lagrangian problem that I encountered during my studies in principal component analysis (PCA).
One should maximize the variance with respect to the first principal component
Following is the case: We know that $W = {\bf v}^T_1 {\bf Sv}_1 $ (eq 1) and that $\bf v_1$ is an orthonormal basis.
Therefore $||{{\bf v}_1 }||^2 = {\bf v}_1^T{\bf v}_1 = 1$ (eq 2). The maximization of $W$ under this constraint can be done by introducing the Lagrangian multiplier λ and maximizing the Lagrangian:
$\mathcal{L} = W + \lambda (1-||{\bf v}_1||^2) = {\bf v}^T_1 ( {\bf S}-\lambda {\bf I}){\bf v}_1 + \lambda$ (eq 3)
So expanding $W$ using eq. 1, and the laws of distributivity we end up with eq 3.
Next, taking the derivative with respect to vector and $\lambda$ we obtain:
$$\frac{\partial}{\partial \lambda} \mathcal{L} = 1-{\bf v}_1^T{\bf v}_1 = 0 $$
$$\nabla_{{\bf V}}{_1}\mathcal{L} = ({\bf S} - \lambda {\bf I}){\bf v}_1 = 0 $$
This can later be shown to be an eigenvalue problem. Can anyone explain to me why the derivatives look like that? I can't seem to figure it out.
Best regards :)
I believe I found a solution, to whom this may be interesting.
First partial differentiation is straight forward, $\lambda$ is a constan. I found a slightly different approach, remember the inner product is = 1.
We know the fact that:$$ \lambda\frac{\partial}{\partial {\bf v}_1^T} ({\bf v}_1^T{\bf v}_1) = 2\lambda {\bf v}_1$$
The second one, uses the fact that $$\frac{\partial}{\partial {\bf v}_1} ({\bf v}_1^T({\bf S}-\lambda {I\bf }){\bf v}_1) = 2({\bf S}-\lambda {\bf I}) {\bf v}_1 $$
Hence the partial differentiation yields:
$$ \nabla _{{\bf v}_1}\mathcal{L} = 2({\bf S}-\lambda {\bf I}) {\bf v}_1 - 2\lambda{\bf v}_1 $$
The optimization problem then becomes: $$ \frac{\partial \mathcal{L}}{\partial{{\bf v}_1}} = ({\bf S}-\lambda {\bf I}) {\bf v}_1 - \lambda{\bf v}_1 = 0$$
Normalization of ${\bf v}_1$ and rewriting of the optimization problem yields the eigenvalue problem: $$ {\bf S v}_1 = \lambda {\bf v}_1 $$