Find the points where the function $f(x,y)$ is continuous where
$$f(x,y)=\left\{ \begin{array}{ll}
\frac{x^2\sin^2y}{x^2+2y^2} & \mbox{if $(x,y)\not=(0,0)$};\\
0 & \mbox{if $(x,y)=(0,0)$}.\end{array} \right.$$
What I attempted: Here $f(x,y)$ is continuous at all the points $(x,y)\not=(0,0)$
We will check continuity at the point $(x,y)=(0,0)$
$f(x,y)=0$ is well defined. Now,
$$\lim_{(x,y)\to (0,0)} \frac{x^2\sin^2y}{x^2+2y^2}$$
Here $0\le \frac{x^2\sin^2y}{x^2+2y^2}\le \frac{x^2y^2}{x^2+2y^2}$ (As $\sin^2y \le y$) [Am I correct here?]
If we let $x=r\cos\theta$, $y=r\sin\theta$, then $$\frac{x^2y^2}{2x^2+y^2}=\frac{r^2 \cos^2\theta r^2 \sin^2\theta}{r^2+r^2 \sin^2\theta}=\frac{r^2 \sin^2\theta \cos^2\theta }{1+\sin^2\theta}\le r^2$$ (As $\frac{\sin^2\theta}{1+\sin^2\theta}\le1$, $\cos^2\theta\le 1$ )
As $(x,y)\to (0,0)$, $r\to 0$. Therefore using Sandwich Theorem we should have $$\lim_{(x,y)\to (0,0)} \frac{x^2\sin^2y}{x^2+2y^2}=0$$
So, it is continuous at $(0,0)$.
I am not sure whether it is correct or not.The question was asked in an interview and I used the above technique. However, they did not pointed my mistake but just gave an another problem of an similar kind where $f(x,y)=\frac{x^2+\sin^2y}{2x^2+y^2}$ for all $(x,y) \not= (0,0)$ and $0$ for the origin.I tried the same method but failed. Then they asked me to leave.
Let $f(x)=\sin^2 x$ and $g(x)=x$. Observe $f(0)=g(0)=0$. Furthermore, $f'(x)=2\sin x \cos x=\sin(2x)$ and $g'(x)=1$. But $f'(x)=\sin(2x) \leq 1=g'(x)$. Therefore, $f(x) \leq g(x)$ for $x \geq 0$. So your inequality is correct.
But this is making things too difficult. This is a Squeeze Theorem problem 'in reverse', meaning while you tend to eliminate the bounded trig function, here you bound the rest. Note that $x^2 \leq x^2 + 2y^2$, we have $\dfrac{x^2}{x^2+2y^2} \leq 1$. Therefore, $$ 0 \leq \dfrac{x^2\sin^2 y}{x^2+2y^2} = \dfrac{x^2}{x^2+2y^2} \cdot \sin^2 y \leq \sin^2y $$ and $\sin^2 y \to 0$ as $(x,y) \to (0,0)$. Therefore by Squeeze Theorem, $\dfrac{x^2\sin^2 y}{x^2+2y^2} \to 0$ as $(x,y) \to (0,0)$. Note that you also want to say that the function is continuous elsewhere. Namely, $x^2 \sin^2 y$ and $x^2+2y^2$ are continuous functions for $(x,y) \neq (0,0)$ and hence $\dfrac{x^2\sin^2 y}{x^2+2y^2}$ is a continuous function for $(x,y) \neq (0,0)$. Therefore, the function $f(x,y)$ is continuous on all of $\mathbb{R}^2$.