Derivative with respect to a vector

Question

I have a function $f: R^{n \times d} \mapsto R$. It's matrix input (say, $G \in R^{n \times d}$) in turn is a linear function of a $d \times 1$ vector $v$. $G = HK$, where $H \in R^{n \times n}$ is a matrix of constants and the first $(n-1)$ rows of $K \in R^{n \times d}$ are constants and the last row is $v^T \in R^{1 \times d}$. I need to find a vector $d \times 1$, which is the derivative of $f$ w.r.t. $v$, i.e., $\frac{\partial f}{\partial v}$. I tried writing it as: $\frac{\partial f}{\partial v} = \frac{\partial f}{\partial G} \frac{\partial G}{\partial v}$. I can get $\frac{\partial f}{\partial G}$, which is $R^{n \times d}$ but I get stuck computing $\frac{\partial G}{\partial v}$. It turns out to be a $n \times d \times d$ tensor. Each $n \times d$ slice of the tensor has only one non-zero column. I am not sure, how to arrive at the final $d \times 1$ (or $1 \times d$) vector, which is $\frac{\partial f}{\partial v}$?

Answer 1

You haven't provided any information about the function $f$ in your problem statement, but I'll assume you know how to calculate its gradient, i.e. $$A=\frac{\partial f}{\partial G}$$ and don't need any help with that term.

To avoid the need to calculate intermediate quantities like $\frac{\partial G}{\partial v}$ (which is a 3rd order tensor by the way), let's proceed using the differential (which is simply a matrix). $$\eqalign{ df &= A:dG \cr &= A:H\,dK \cr &= H^TA:dK \cr \frac{\partial f}{\partial K} &= H^TA = H^T\frac{\partial f}{\partial G} \cr }$$ We have found the gradient wrt the entire $K$ matrix, but you only wanted to know about the last row. Multiply the result by the cartesian basis vector $e_n$ since $(v^T = e_n^TK)$ yields $$\eqalign{ \frac{\partial f}{\partial v} &= e_n^T\frac{\partial f}{\partial K} = e_n^TH^T\frac{\partial f}{\partial G} \cr }$$ In some of the steps above, I used a colon to denote the trace/Frobenius product $$A:B={\rm tr}(A^TB)$$