Consider the follow equation
$$f(\vec{x}) = c(\vec{x})B\vec{x}$$
where $c(\vec{x})$ is given by the generalized Rayleigh quotient
$$c(\vec{x}) = \frac{\vec{x}^{T}A\vec{x}}{\vec{x}^{T}B\vec{x}}$$
$\vec{x}$ is a vector, and A and B are square matrices. I want to take the derivative of this equation with respect to $\vec{x}$. My intuition is to use the product rule to obtain
$$\frac{\partial f(\vec{x})}{\partial \vec{x}} = c'(\vec{x})A \vec{x} + c(\vec{x})B.$$
I know that the derivative of $c(\vec{x})$ is given by
$$c'(\vec{x}) = \frac{2(\vec{x}^{T}B\vec{x})A\vec{x} - 2(\vec{x}^{T}A\vec{x})B\vec{x}}{(\vec{x}^{T}B\vec{x})^{2}}$$
which is a vector. However, I do not believe the product of this derivative and the matrix, $c'(\vec{x})A\vec{x}$, to be valid since this would be a scalar and I would be adding it to a matrix. How does one compute this derivative? Any references would be welcomed.
$f({\bf x})$ is a vector, you can write its $i$-th component as
$$ f_i({\bf x}) = c({\bf x})(B {\bf x})_i = c({\bf x})\sum_k B_{ik} x_k $$
The derivative of $f_i({\bf x})$ w.r.t to $x_j$ is
\begin{eqnarray} \frac{\partial f_i({\bf x})}{\partial x_j} &=& \frac{\partial c({\bf x})}{\partial x_j} (B{\bf x})_i + c({\bf x})\frac{\partial}{\partial x_j}\sum_k B_{ik}x_k \\ &=& c'_j({\bf x}) (B{\bf x})_i + c({\bf x})\sum_k B_{ik}\frac{\partial x_k}{\partial x_j} \\ &=& c'_j({\bf x}) (B{\bf x})_i + c({\bf x}) B_{ij} \end{eqnarray}
So the result of calculating the gradient of $f({\bf x})$ is the matrix
$$ \frac{\partial f({\bf x})}{\partial {\bf x}} = B{\bf x} [c'({\bf x})]^T + c({\bf x}) B $$