Why is the inner product of a quadratic form a quadratic form?

Question

I was going through a derivation of the second derivative of the $\log \det X$ where $X$ is symmetric positive definite, I noticed that despite the second order approximation of log det is written as:

$f(Y) \approx f(X) + \langle \nabla f, x_o \rangle + \frac{1}{2}\langle \nabla^2 f, x_o \rangle$

The explicit second order approximation of log det is written as:

$f(Y) \approx f(X) + \nabla f^T x_o+ \frac{1}{2} x_o^T \nabla^2 f x_o$

where $x_o = Y - X$

If I recall correctly, the inner product $\langle A ,B \rangle$ = $A^TB$ Can someone explain why it changes into $\langle A ,B \rangle$ = $B^TAB$ when $A$ is a quadratic form?

Answer 1

You seem to be mixing ideas up in the last paragraph. I'll just state those as clearly as I can and hope that helps.

In general, any bilinear form in a finite dimensional vector space $V$ with a given basis has a "Gram matrix" $M$ such that $\langle x,y \rangle=xMy^\top$ for $x,y\in V$, everything written in terms of the basis.

If the bilinear form is a symmetric positive definite real form, then you can always change basis in $V$ so that $M$ becomes the identity matrix, and then you just have $\langle x,y \rangle=xy^\top$ for $x,y\in V$.

You automatically get a quadratic form by using $x=y$ so that $Q(x):=xMx^\top$.

Answer 2

1) The general Taylor formula is wrong. You have $\langle x_o, \nabla^2 f(X) x_o \rangle$.

2) Your inner product for matrices is wrong. It is $\langle X, Y \rangle = \operatorname{tr}(X^T Y) = \operatorname{tr}(X Y^T)$.