Derivative of $(Ax - y)^T(Ax - y)$ where $x$ and $y$ are vectors of $n$ dimensions and $A$ is matrix $n\times n$

Question

Derivative of $(Ax - y)^T(Ax - y)$ where $x$ and $y$ are vector of $n$ dimension and $A$ is matrix $n\times n$

I solved the function and broke it down to following expression:

$$f = x^TA^TAx -2x^TA^Ty + y^Ty$$

When taking its derivative $df/dx$ I am applying product rule to first and reaching to.

$$x^TA^TA + A^TAx - 2 A^Ty$$

Now the problem is this answer in not in any of the given options. Can anyone please tell me what am I doing wrong?

Answer 1

The dimension of $x^TA^TA$ and $A^TAx$ don't match.

$$A^TAx + A^TAx-2A^Ty = 2A^TAx - 2A^Ty$$

Answer 2

Please refer to the book: The matrix cookbook

And then apply to your problem: \begin{align} \frac{\partial f}{\partial x} =& \frac{\partial (x^\top A^\top Ax - x^\top A^\top y - y^\top Ax + y^\top y)}{\partial x}\\ =& A^\top Ax + A^\top Ax - A^\top y- A^\top y\\ =& 2 A^\top Ax - 2A^\top y \end{align} And moreover, \begin{align} \frac{\partial f}{\partial y} =& -Ax-Ax+2y\\ =& -2Ax + 2y \end{align}

It is worth to point that, since your function is a $\mathbb{R}^n \Rightarrow \mathbb{R}^1$, which means the result is a real value. Then it could apply the formula. If the result is a vector or matrix, it will become more complicated.