Gradient and Hessian of this function

Question

What is the gradient $\nabla_x$ / hessian $\nabla_x^2$ of $\frac{(Ax)^{\top}y}{||Ax||}$ where $A \in \mathbb{R}^{m \, \times \, n}$, $x \in \mathbb{R}^n$ and $y \in \mathbb{R}^m$?

I don't really know how to approach this problem because I don't know how to deal with quotients in matrix calculus.

Answer 1

For convenience, let's define some variables $$\eqalign{ b &= Ax &\implies &db = A\,dx \cr 1 &= \alpha\beta &\implies &d\alpha = -\alpha^2d\beta \cr \beta^2 &= b^Tb &\implies &\beta\,d\beta = b^Tdb=b^TA\,dx \cr &\, &\, &d\beta = \alpha\,b^Tdb=\alpha\,b^TA\,dx = z^TA\,dx \cr z &= \alpha b &\implies &dz =\alpha\,db-b\alpha^3(b^Tdb)=\alpha(I-zz^T)A\,dx \cr \cr}$$ Write the function in terms of these variables and find its differential and gradient $$\eqalign{ f &= y^Tz \cr\cr df &= y^Tdz \cr &= y^T\alpha(I-zz^T)A\,dx \cr &= \alpha(y^T-fz^T)A\,dx \cr \cr g^T=\frac{\partial f}{\partial x^T} &= \alpha(y-fz)^TA \cr\cr }$$ Now find the differential and gradient of the gradient $$\eqalign{ dg^T &= d\alpha(y-fz)^TA -df\,\alpha z^TA -\alpha f\,dz^TA \cr &= -\alpha\,d\beta\,g^T -\alpha(dx^Tg)z^TA -\alpha^2f\,dx^TA^T(I-zz^T)A \cr &= -\alpha\,dx^T\Big(A^Tzg^T +gz^TA +\alpha f(A^TA-A^Tzz^TA)\Big) \cr\cr H = \frac{\partial g^T}{\partial x} &= -\alpha\Big(A^Tzg^T +gz^TA +\alpha f(A^TA-A^Tzz^TA)\Big) \cr \cr}$$ This can be made nicer by introducing yet more variables $$\eqalign{ M &= \alpha A,\,\,\,\,\,\, p = M^Tz \cr }$$ yielding $$\eqalign{ g &= M^Ty-fp \cr H &= fpp^T - pg^T - gp^T - fM^TM \cr }$$

Answer 2

Strategy to approach problem:

1. Let $g(x) = (Ax)^Ty$ and $h(x) = \|Ax\|$, then the function you want to differentiate is $f(x) = g(x)h(x)^{-1}$.
2. Recall that $\nabla_x = (\partial_1, \partial_2, \dots \partial_n)$ where we use the shorthand $\partial_i = \partial/\partial x^i$, so it suffices to compute $\partial_if = (\partial_i g) h^{-1} + g\partial_i(h^{-1}) = (\partial_i g) h^{-1} -g(\partial_ih)/h^2$.
3. It remains to compute $\partial_i g$ and $\partial_i h$. This is easiest if one writes these functions in index notation (especially if one uses the Einstein summation notation to avoid summation symbols). For example $$ \partial_i g = \partial_i[(Ax)^jy_j] = \partial_i(A^j_kx^ky_j) = A^i_ky_j\partial_ix^k=A^j_ky_j\delta^k_i = A^j_iy_j $$

The Hessian is more work since in index notation is a matrix of mixed partial derivatives, but essentially the same strategy works for computing an arbitrary matrix entry.