Jacobian and vectorization

Question

Given the matrix function $h(Q) = Q^{T}AQ$.

The derivative can be obtained as

$$\lim_{\epsilon \to 0} \frac{h(Q - \epsilon H)-h(Q)}{\epsilon} = H^{T} A Q + Q^{T} A H$$

Then, I saw that the Jacobian $J_h(vec(Q)) = ((AQ)^{T} \otimes I)\Pi + I \otimes Q^{T} A$. I have some issues with obtaining this identity. Notice that $\Pi$ is the matrix s.t. $\Pi vec(X) = vec(X^{T}) \ \forall X$.

I tried this:

\begin{align*} vec(H^{T} A Q + Q^{T} A H) &= vec(H^{T} A Q) + vec(Q^{T} A H)\\ &= (Q^{T} \otimes H^{T}) vec(A) + (H^{T} \otimes Q^{T}) vec(A)\\ &= (Q^{T} \otimes I) (I \otimes H^{T}) vec(A) + (H^{T} \otimes I)(I \otimes Q^{T})vec(A)\\ &= (Q^{T} \otimes H^{T}) vec(A) + (H^{T} \otimes Q^{T})vec(A)\\ &= ((Q^{T} \otimes H^{T}) + (H^{T} \otimes Q^{T})) vec(A). \end{align*}

I am afraid that I am missing up something in the beginning, since there's no need for me to use $\Pi$. Any advice is appreciated.

Answer 1

Note that a function with a matrix input and matrix output does not, technically speaking, have a Jacobian. What it seems that you're actually interested in here is the Jacobian of the related function $f$ that takes a vector input and vector output such that $f(\operatorname{vec}(Q)) = \operatorname{vec}(h(Q))$. That is, you're interested in the Jacobian of the "vectorized version" of $h$.

In order to go from the Frechet/total derivative of a function $f:\Bbb R^n \to \Bbb R^m$ to a Jacobian, you need to write the function $df(q):\Bbb R^n \to \Bbb R^m$ (for $q \in \Bbb R^n$) in the form $M(q) \Delta q$ for some matrix $M$ depending on $q$. This matrix $M(q)$ is the Jacobian of the function $f$.

In other words, the relationship between the Jacobian of $f$ and the Frechet/total derivative of $f$ is that $$ df(x)(h) = J_f(x)h, \quad x,h \in \Bbb R^n. $$ Note that $h$ in the above is a vector rather than a function.

In your case, you have found that $$ dh(Q)(\Delta Q) = \Delta Q^TAQ + Q^TA\Delta Q. $$ If we define $f$ so that $f(\operatorname{vec}(Q)) = \operatorname{vec}(h(Q))$, then it follows that \begin{align} df(\operatorname{vec}(Q))(\operatorname{vec}(\Delta Q)) &= \operatorname{vec}(\Delta Q^TAQ + Q^TA\Delta Q) \\ & = \operatorname{vec}(\Delta Q^TAQ) + \operatorname{vec}(Q^TA\Delta Q) \\ & = [(AQ)^T \otimes I] \operatorname{vec}(\Delta Q^T) + [I \otimes (Q^TA)] \operatorname{vec}(\Delta Q) \\ & = [(AQ)^T \otimes I] \Pi\operatorname{vec}(\Delta Q) + [I \otimes (Q^TA)] \operatorname{vec}(\Delta Q) \\ & = \left[((AQ)^T \otimes I) \Pi + (I \otimes Q^TA)\right]\operatorname{vec}(\Delta Q) \end{align} Replacing $\operatorname{vec}(Q)$ with $q$ and $\operatorname{vec}(\Delta Q)$ with $\Delta q$ in the above gives us $$ df(q)(\Delta q) = \left[((AQ)^T \otimes I) \Pi + (I \otimes Q^TA)\right] \Delta q. $$ Thus, the Jacobian of $f$ (the vectorized version of $h$) is given by $$ J_f = ((AQ)^T \otimes I) \Pi + (I \otimes Q^TA). $$

Answer 2

By substituting I brute forced my way to the identity. Although I am not sure why I am supposed to apply the perfect shuffle matrix and then divide by $vec(H)$.

From $H^{T} A Q + Q^{T} A H$, then put $\kappa = AQ$ and $\lambda = Q^{T}A$ Then:

\begin{align*} vec(H^{T} \kappa) + vec(\lambda H) &= (\kappa^{T} \otimes I) vec(H^{T}) + (I \oplus \lambda) vec(H)\\ &= ( \kappa^{T} \otimes I) \Pi vec(H) + (I \otimes \lambda) vec(H) \end{align*} Then apparently one has to divide by $vec(H)$ to obtain

\begin{align*} (\kappa^{T} \otimes I) \Pi + (I \otimes \lambda) = ((AQ)^{T} \otimes I) \Pi + I \otimes Q^{T} A \end{align*}

Answer 3

Using the more standard $K$ $($instead of $\Pi)$ to denote the Commutation Matrix, an uppercase $H$ to denote your $h$ matrix, lowercase letters $(h,q)$ to denote the vectorized form of the matrices $(H,Q),\,$ and using differentials instead of limits, the calculation runs as follows $$\eqalign{ \def\p{\partial} \def\k{\otimes} \def\v{\operatorname{vec}} H &= Q^TAQ \\ dH &= dQ^TAQ \;+\; Q^TA\,dQ \\ \v(dH) &= ((AQ)^T\k I)\v(dQ^T) \;+\; (I\k Q^TA)\v(dQ) \\ dh &= (Q^TA^T\!\k I)K\,dq \;+\; (I\k Q^TA)\,dq \\ \frac{\p h}{\p q} &= (Q^TA^T\!\k I)K \;+\; (I\k Q^TA) \\ }$$