How do I find the Hessian (second-derivative) of $f(X) = tr(A X^T B X C)$?
Using differential notation, this has the following first differential:
$d tr(A X^T B X C) = tr((CAX^TB + A^T C^T X^T B^T) dX) $.
This straightforwardly leads to the first derivative $(CAX^TB + A^T C^T X^T B^T)^T = B^T X A^TC^T + BXCA$.
If I've done my algebra correctly, the second differential is
$ d^2 tr(A X^T B X C) = tr(CAdX^T B dX + A^TC^TdX^T B^T dX)$.
How do I obtain the Hessian from this? I believe I need to somehow put this into the form $vec(dX)^T H vec(dX)$, from which I can conclude that the Hessian is $H$.
$ \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\vc#1{\op{vec}\LR{#1}} \def\trace#1{\op{Tr}\LR{#1}} \def\frob#1{\left\| #1 \right\|_F} \def\qiq{\quad\implies\quad} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} $Your gradient calculation is indeed correct. To find the Hessian...
First flatten $X$ and the gradient matrix into vectors $$\eqalign{ x &= \vc{X} \\ G &= B^TXA^TC^T + BXCA \\ g &= \vc{G} \\&= \LR{CA\otimes B^T}x + \LR{A^TC^T\otimes B}x \\ }$$ Then calculate the gradient of the gradient (i.e. the Hessian) $$\eqalign{ dg &= \LR{CA\otimes B^T}dx + \LR{A^TC^T\otimes B}dx \\ \grad{g}{x} &= \LR{CA\otimes B^T} + \LR{A^TC^T\otimes B} \;\equiv\; Hessian \\ }$$