Let ${\bf{x}}\in \mathbb{R}^{n}$ be a column vector: $${\bf{x}} = \begin{pmatrix} x_{1} \\ \vdots \\ x_{n} \end{pmatrix} $$ and ${\bf{y}} \in \mathbb{R}^{n}$ analogously. Let $A$ and $B$ be two $n \times n$ matrices and consider the following objects: $$Q({\bf{x}},{\bf{y}}) \equiv Q = {\bf{y}}^{T}A{\bf{x}} \quad \mbox{and} \quad Z({\bf{x}},{\bf{y}}) \equiv Z = ({\bf{y}}^{T}B{\bf{x}})^{2}$$ Moreover, let $S$ be a third $n \times n$ matrix. I want to calculate the following object: $$\sum_{i,j=1}^{n}\frac{\partial Q}{\partial y_{i}}S_{ij}\frac{\partial Z}{\partial x_{j}}$$
Applying the chain rule I found: $$\sum_{i,j=1}^{n}\frac{\partial Q}{\partial y_{i}}S_{ij}\frac{\partial Z}{\partial x_{j}} = ({\bf{y}}^{T}B{\bf{x}})({\bf{y}}^{T}A^{T}SB{\bf{x}})$$
Is my formula correct?
We can write $Q=\sum_{i,j=1}^n A_{ij}y_ix_j$ and $Z=\left(\sum_{i,j=1}^n B_{ij}y_ix_j\right)^2$. Hence we have $$\frac{\partial Q}{\partial y_i} = \sum_{k=1}^n A_{ik}x_k$$ and $$\frac{\partial Z}{\partial x_j} = 2(\mathbf{y}^TB\mathbf{x})\sum_{l=1}^n B_{lj}y_j.$$ Thus $$\sum_{i,j=1}^n \frac{\partial Q}{\partial y_i}S_{ij}\frac{\partial Z}{\partial x_j}=\sum_{i,j,k,l=1}^n 2(\mathbf{y}^TB\mathbf{x})A_{ik}x_kS_{ij}B_{lj}y_l =2(\mathbf{y}^TB\mathbf{x})(\mathbf{y}^TBS^TA\mathbf{x}).$$ Thus, the proposed formula $(\mathbf{y}^TB\mathbf{x})(\mathbf{y}^TA^TSB\mathbf{x})$ in the original post is not correct.
Note that both $\mathbf{y}^TB\mathbf{x}$ and $\mathbf{y}^TBS^TA\mathbf{x}$ are scalars. Thus we have $\mathbf{y}^TB\mathbf{x} = (\mathbf{y}^TB\mathbf{x})^T = \mathbf{x}^TB^T\mathbf{y}$, and similarly we also have $\mathbf{y}^TBS^TA\mathbf{x}=(\mathbf{y}^TBS^TA\mathbf{x})^T=\mathbf{x}^TA^TSB^T\mathbf{y}$.