Let $X,Y$ be Hilbert spaces, and let $A\colon X\to Y$ be a compact operator. The Tikhonov functional is given by $$ F(x)=\lVert Ax-y\rVert_X^2+\alpha\lVert x\rVert_X^2. $$ Calculate the Gâteaux derivative of the Tikhonov functional.
Tip: Use $\lVert a+b\rVert=\lVert a\rVert^2+2(a,b)+\lVert b\rVert^2$.
First question: Is it really $\lVert Ax-y\rVert_X^2$ and not the Y-norm?
Second question: What I have to do here is to my opinion calculate
$$ \lim\limits_{t\to 0}\frac{F(x+th)-F(x)}{t}. $$
So I would start with calculating $F(x+th)=\lVert A(x+th)-y)\rVert_X^2+\alpha\lVert x+th\rVert_X^2$.
I start with calculating $\lVert A(x+th)-y)_X^2$. I can not read from the text if the operator $A$ is linear, too. Is it right to calculate with the given tip
$$\lVert A(x+th)-y\rVert_X^2=\lVert A(th)+Ax-y\rVert_X^2=\lVert A(th)\rVert_X^2+2(A(th),Ax-y)+\lVert Ax-y\rVert_X^2?$$