Question: What is a compact notation for expressing the quantity?
$$ \frac{\partial}{\partial x_i} D^2 \mathbf{F}( \mathbf{g}(x) ) \mathbf{h}(x) \tag{Q} $$
To be concrete and keep things simple (clearly this can be generalized to other dimensions), let's suppose: $x = (x_1, \ldots, x_n)$, $\mathbf{F}( \mathbf{z} ) = \mathbf{F}(z_1, z_2)$, $\mathbf{g}(x) = ( g_1(x), g_2(x) )$, and $\mathbf{h}(x) = (h_1(x), h_2(x))$. Thus, we have the Hessian as,
$$ D^2 \mathbf{F}(z) = \begin{bmatrix} \partial^2 F / \partial z_1 \partial z_1 & \partial^2 F / \partial z_1 \partial z_2 \\ \partial^2 F / \partial z_1 \partial z_2 & \partial^2 F / \partial z_2 \partial z_2 \end{bmatrix} =: \begin{bmatrix} F_{11} & F_{12} \\ F_{12} & F_{22} \end{bmatrix} $$ Hence, we know that $D^2 \mathbf{F}( \mathbf{g}(x) ) \mathbf{h}(x)$ is a $(2 \times 2)(2 \times 1) = 2 \times 1$ vector, and thus $\frac{\partial}{\partial x_i} D^2 \mathbf{F}( \mathbf{g}(x) ) \mathbf{h}(x)$ is also a $2 \times 1$ vector.
The difficultly that I'm having is the following. If I brute force expand out $D^2 \mathbf{F}( \mathbf{g}(x) ) \mathbf{h}(x)$ and just take derivative $\partial / \partial x_i$ element-by-element and apply the product rule, I end up with a quantity that looks like:
$$ \begin{align} \frac{\partial}{\partial x_i} D^2 \mathbf{F}( \mathbf{g}(x) ) \mathbf{h}(x) &= D^2 \mathbf{F}( \mathbf{g}(x) ) \frac{\partial \mathbf{h}(x)}{\partial x_i} + \left( \frac{\partial}{\partial x_i} D^2 \mathbf{F}( \mathbf{g}(x) ) \right) \mathbf{h}(x) \end{align} $$
My trouble is with writing a compact notation for $\frac{\partial}{\partial x_i} D^2 \mathbf{F}( \mathbf{g}(x) )$. Writing the terms out, I see that, $$ \begin{align} \frac{\partial}{\partial x_i} D^2 \mathbf{F}( \mathbf{g}(x) ) &= \begin{bmatrix} \partial F_{11}(\mathbf{g}) / \partial x_i & \partial F_{12}(\mathbf{g}) / \partial x_i \\ \partial F_{12}(\mathbf{g}) / \partial x_i & \partial F_{22}(\mathbf{g}) / \partial x_i \end{bmatrix} \end{align} $$ Applying the chain rule say for the first element (similar for the other 3 elements), I see that,
$$ \begin{align} \frac{\partial}{\partial x_i} F_{11}(\mathbf{g}) &= \frac{\partial F_{11}}{\partial z_1} \frac{\partial g_1}{\partial x_i} + \frac{\partial F_{11}}{\partial z_2} \frac{\partial g_2}{\partial x_i} \\ &=: ( \nabla_z F_{11} )^\top ( \nabla_{x_i} \mathbf{g} ) \end{align} $$ which means the above has the form,
$$ \frac{\partial}{\partial x_i} D^2 \mathbf{F}( \mathbf{g}(x) ) = \begin{bmatrix} ( \nabla_z F_{11} )^\top ( \nabla_{x_i} \mathbf{g} ) & ( \nabla_z F_{12} )^\top ( \nabla_{x_i} \mathbf{g} ) \\ ( \nabla_z F_{12} )^\top ( \nabla_{x_i} \mathbf{g} ) & ( \nabla_z F_{22} )^\top ( \nabla_{x_i} \mathbf{g} ) \end{bmatrix} \tag{1} $$
* In all: * Is there a compact notation for expressing the right hand side of (1), which then solves my question to (Q)?