I am more interested in the method which I use for it to be done. I need to understand all the in-between steps so that I can apply it to other examples too.
Given the functional:
\begin{equation} f(x,y)=\begin{cases} \frac{x^3y}{x^4+y^2} & x \neq 0, y\neq 0 \\ 0, & x=y=0\end{cases} \end{equation}
it is Gateaux differentiable (why?) and its Gateaux derivative at 0 is equal to 0. Moreover:
\begin{equation} \frac{|f(x,x^2)|}{\|(x,x^2)\|}=\frac{1}{2 \sqrt{1+x^2}}\to \frac{1}{2}, \quad x \to 0 \end{equation}
which means that $f$ is not Frechet differentiable at $(0,0)$.
How do I see that from the last relationship? I know that this is trivial but I need to make everything clear in my mind so that I can move forward.
Thank you for your time!
The Gâteaux derivative at a point $x$ in the direction $h$ is given by $df(x,h) = \lim_{t \to 0} {f(x+th)-f(x) \over t}$.
Since $f(0,0) = 0$, we have, by computation, $d f((x,y), h) = \lim_{t \to 0} {1 \over t}({ h_1^3 h_2 t^4 \over h_1^4 t^4+h_2^2t^2} ) = 0$.
If $f$ had a Fréchet derivative at $x$, then $df(x,h) = Df(x)h$, where $Df(x)$ is the Fréchet derivative of $f$.
In particular, the above would imply that $Df((0,0)) = 0$, if $f$ was Fréchet differentiable at $(0,0)$. In particular, for any sequence $(x_n,y_n) \to (0,0)$, we would have $\lim_n { |f(x_n,y_n)- f(0,0)- 0 (x_n,y_n)^T| \over \|(x_n,y_n)\|} = \lim_n { |f(x_n,y_n)| \over \|(x_n,y_n)\|} = 0$.
However, the path in the question shows that taking $x_n = {1 \over n}, y_n = {1 \over n^2}$ results in a different limit. Hence $f$ cannot be Fréchet differentiable at $(0,0)$.