I am stuck computing the derivate of the function
$$ h(\vec{x}, \vec{p}, \vec{q}_c, \vec{t}) = \xi \; \omega \; {|| R(\vec{q}_c)\vec{x} + \vec{t} - \vec{p} ||}^2 $$ where $R(\vec{q}_c) \in \mathbb{R}^4 \times \mathbb{R}^4$ represents the $4 \times 4$ rotation matrix. $\vec{x}, \vec{p}, \vec{t} \in \mathbb{R}^4$, $\omega, \xi \in \mathbb{R}$ , $\vec{q}_c$ is a quaternion, and $$ \frac{\partial}{\partial q} R(\vec{q}_c)\vec{x} = 2 C(\vec{x})^T $$ where $C(\vec{x})$ is the skew-symmetric matrix of $\vec{x}$.
I know that $$ \frac{\partial}{\partial \vec{x}} ||\vec{x}||^2 = 2 \vec{x} $$ and I have derived that $$ \frac{\partial}{\partial \vec{x}} ||f(x)||^2 = 2 \cdot \text{diag}(f(x)) \cdot \frac{\partial}{\partial x} f(x) $$ which would mean $$ \frac{\partial h}{\partial \vec{q}} = \xi \; \omega \; 2 \; \text{diag}(R(\vec{q}_c)\vec{x} + \vec{t} - \vec{p} ) 2 C(\vec{x})^T $$
however since $\frac{\partial h}{\partial \vec{q}} \in \mathbb{R}^4 \times \mathbb{R}^4$ instead of $\in \mathbb{R}^4$ I strongly suspect my result to be incorrect.
Can anyone point me in the right direction or tell me where I am going wrong.
(I lack repuation to ask for clarifications in a comment. Sorry about that)
I am assuming that by $\mathbb{R}^4 \times \mathbb{R}^4$, you mean 4-by-4 real matrices (as opposed to pairs of vectors of length 4). This way, the product $R(q_c) x$ makes sense :)
The derivative of $\|f(x)\|^2$ is indeed not correct; if $f:\mathbb{R}^4\to\mathbb{R}^4$ we would expect $\frac\partial{\partial x}\|f(x)\|^2$ to be a vector, but $\frac\partial{\partial x}f(x)$ is a matrix, and so is $\text{diag}(f(x)) \frac\partial{\partial x}f(x)$.
An easy way to find the correct derivative, is to write the squared norm of $f$ as the product $f(x)^\top f(x)$. This allows you to use the product rule, which dictates $$\frac\partial{\partial x} (f(x)^\top f(x)) = \frac{\partial f(x)}{\partial x}^\top f(x) + \left(f(x)^\top \frac{\partial f(x)}{\partial x}\right)^\top = 2 \frac{\partial f(x)}{\partial x}^\top f(x).$$
In your specific case, this would mean $$\frac{\partial h}{\partial q} = 4 \,\xi \,\omega \,C(x) (R(q_c)\,x + t - p).$$