I'm studying the continuity of the function $$ f(x,y) = \left\{ \begin{array}{l l} \frac{x^2y^2}{x^2+y^2} & \quad , \quad(x,y)\neq(0,0)\\ 0 & \quad , \quad(x,y)=(0,0) \end{array} \right.$$
in the point $(x,y)=(0,0)$.
It's clear to me that if a function is not continuous I have to find a case of discontinuity, but perhaps it's more difficult to prove the continuity in which I can't find a fault in the behaviour of the function.
In this case how can I show the limit $ \lim_{(x,y)\rightarrow (0,0)} f(x,y) = \lim_{(x,y)\rightarrow (0,0)} \frac{x^2y^2}{x^2+y^2} =0$?
It's simple to prove that on a line $y=mx$ through the origin $(0,0)$ but in the most general way?
For $x=0$ or $y=0$, it is clear that the fraction equals $0$, so we may, without loss of generality, assume that we approach the origin through a path where $x,y\neq0$. Then we can apply AM-GM to observe that $x^2+y^2\geq2\sqrt{x^2y^2}=2|xy|$. This yields $$\lim_{(x,y)\rightarrow (0,0)} \left|\frac{x^2y^2}{x^2+y^2}\right|\leq\lim_{(x,y)\rightarrow (0,0)} \left|\frac{x^2y^2}{2|xy|}\right|=\lim_{(x,y)\rightarrow (0,0)}\frac{|xy|}{2}=0.$$
This is just a nice trick that often works. It can be done way simpler in this case, however: $$\lim_{(x,y)\rightarrow (0,0)} \left|\frac{x^2y^2}{x^2+y^2}\right|\leq\lim_{(x,y)\rightarrow (0,0)} \left|\frac{x^2y^2}{x^2}\right|=\lim_{(x,y)\rightarrow (0,0)} |y^2|=0.$$
There are many strategies to tackle these problems, and it is probably best to try the second approach first. AM-GM is a nice trick that often works.
It is really important, however, that the fact that the iterated limit, for an $m\in\mathbb{R}$ $$\lim_{x\to0}\lim_{y\to mx}f(x,y)=L,$$ does not imply the limit exists. For example, the limit of $f(x,y)=\frac{x^2y}{x^4+y^2}$ as $(x,y)\to(0,0)$ exists along every straight line approaching the origin but not along the parabola $y=x^2$.