I am struggling to derive this as I am still getting my head around matrix manipulation.
How would I simplify this least square algebraically i.e I am looking for $(a+b)^2 = a^2+2ab +b^2$ equivalent of this: $$\Vert y - A(A^TA)^{-1} (\lambda+A^Ty)\Vert_2^2$$
where $A \in \mathbb{R}^{M\times N}$, $y \in \mathbb{R}^M$ and $\lambda \in \mathbb{R}^N$ and $A$ is full Rank.
Edit: Correction $\lambda \in \mathbb{R}^N$, (previously $\lambda \in \mathbb{R}$).
This problem is arising from a non-negative Least Squares problem, where I started with the Lagrangian: $$ \min_{x \in \mathbb{R}^{N}} \Vert y-Ax \Vert_2^2 -\lambda^Tx$$ Whose Solution is: $$x^* = (A^TA)^{-1} (\lambda+ A^Ty)$$ Plugging this back into the Lagrangian to find the dual results an expression a part of which is the expression in my question. Just to give background.
Assuming that you mean $y - A(A^TA)^{-1}(\lambda I + A^T y)$, we can rewrite it as $$Iy - A(A^TA)^{-1}(\lambda I + A^T y) = (I-A(A^TA)^{-1}A^T)y - \lambda A(A^TA)^{-1} = My -N$$
Now we have $$\|My-N\|_2^2$$ which in some sense is a standard form we can work with
The only difference with the knowledge $\lambda$ is a matrix, let us call it $\Lambda$ instead, will be that we need to make sure we respect that matrix multiplication is not commutative, so when multiplying together we get $$A(A^TA)^{-1}\Lambda$$ with $\Lambda$ to the right so the expression becomes
$$(I-A(A^TA)^{-1}A^T)y - A(A^TA)^{-1}\Lambda$$
It will still be on the same form $My-N$, just that the $N$ matrix is slightly different.