Derivative of transpose

Question

I am trying to find derivative of this : RQ(u) = u^TX^TXu / u^Tu

I need help finding derivative : RQ/u

Optimal sol should satisfy X^TXu = RQ(u)u

I am very confused, any help would be great. If you could share some references that will be very helpful too.

Answer 1

The quantity you are interested in is known as the Rayleigh Quotient. We have

$$ \frac{\partial}{\partial u} \frac{u^T X^T X u}{u^T u} = \frac{1}{u^T u} \frac{\partial}{\partial u}(u^T X^T X u) + (u^T X^T X u) \frac{\partial}{\partial u}(u^T u)^{-1} $$

Now use $\frac{\partial}{\partial u} u^T A u = 2Au$ for symmetric matrix $A$: $$ \implies\frac{\partial}{\partial u} RQ(u) = \frac{2X^T X u }{u^T u} - \frac{ (u^T X^T X u) 2 I u}{(u^T u)^2} = \frac{2}{u^T u }\big(X^T X - RQ(u)\cdot I\big)u $$

In particular, if $u$ is a maximizer/minimizer of $RQ$ then

$$ 0 = \frac{\partial}{\partial u} RQ(u) \implies \big(X^T X - RQ(u)\cdot I\big)u =0 $$

If you are unsure about the matrix derivatives, I recommend this online tool: http://www.matrixcalculus.org/, and for fomulas wikipedia and The Matrix Cook Book

Answer 2

We have $ y = u^T X^T X u $. We compute the differential of $ y$ $$ dy = (du)^T X^T X u + u^T X^T X (du) \\ dy = Tr ((du)^T X^T X u + u^T X^T X (du)) \\ dy = Tr ( u^T X^T X (du)) + Tr( u^T X^T X (du) ) \\ dy = 2 Tr ( u^T X^T X (du)) \\ dy = 2 (u^T X^T X)^T: du $$ Finally $dy/du = 2 (u^T X^T X)^T = 2 X^T X u $

Additional information
Here I have used two properties of the trace:
1) $ Tr(ABC) = Tr(BCA) = Tr(CAB) $
2) $ Tr(c) = c $, where $c$ is an scalar