which one is the correct derivative w.r.t time?

Question

We want to take time derivative of

$x^TPx$

and we know $\dot{x}=Ax,~~P^T=P$.

Which one is wrong and why?

1- $d(x^TPx)/dt=\nabla (x^TPx).\dot{x}=2x^TP\dot{x}=2x^TPAx.$

2- $d(x^TPx)/dt=\dot{x}^TPx+x^TP\dot{x}=x^TA^TPx+x^TPAx$

Answer 1

The second is correct - keyword "chain rule". The first is wrong because it differs.

Answer 2

The second expression is correct, and can be recast into a form that looks more like the first, i.e. $\,\,2\,x^TA^TPx $

You just got your matrix term transposed somehow.

To prevent myself from making transposition mistakes, I usually employ a derivation in terms of the Frobenius product, like so
$$\eqalign{ f &= x^TPx \cr &= P:xx^T \cr\cr df &= P:2\,{\rm sym}(dx\,x^T) \cr &= 2\,{\rm sym}(P):dx\,x^T \cr &= 2\,Px:dx \cr &= 2\,Px:Ax\,dt \cr\cr \frac{df}{dt} &= 2\,Px:Ax \cr &= 2\,A^TP:xx^T \cr &= 2\,x^TA^TPx \cr }$$

where $\,{\rm sym}(M) \equiv \frac{1}{2}(M+M^T)\,$ and $\,A\!:\!B={\rm tr}(A^TB)$ is the Frobenius product.