Determine whether the function $f : \mathbb{R}^2 \to \mathbb{R}$ defined as: $$\begin{cases}\frac{xy^3}{x^2+y^6} \qquad (x,y)\neq (0,0) \\ 0 \quad \ \ \ \qquad (x,y)=(0,0) \end{cases}$$ is differentiable.
That's how I proceded. Clearly $f$ is differentiable in $\mathbb{R}^2 \setminus \{0\}$ simply by seeing that the partial derivatives are continuous in this set. The only problem is in the origin. Let's show that the function is differentiable in $(0,0)$ by applying the definition: we want that $$\lim_{(h,k)\to (0,0)}\frac{f(h,k)-f(0,0)-<\nabla(f)(0,0),(h,k)>}{\sqrt{(h^2+k^2)}}=0$$ As we have $f(0,0)=0$ we have that $f_x(0,0)=f_y(0,0)=0$ and so the scalar product cancels out. We now have to compute:
$$\lim_{(h,k)\to(0,0)}\frac{hk^3}{(h^2+k^6)\sqrt{(h^2+k^2)}}$$
By using polar coordinates we have:
$$\lim_{(h,k)\to(0,0)}\left|\frac{hk^3}{(h^2+k^6)\sqrt{(h^2+k^2)}} \right|\leq \lim_{(h,k)\to(0,0)}\left|\frac{hk^3}{h^2\sqrt{(h^2+k^2)}} \right|=\lim_{\rho \to 0} \left| \frac{\rho\cos\theta\rho^3\sin^3\theta}{\rho^3\cos^2\theta}\right|=\lim_{\rho \to 0}\left|\frac{\rho\sin^3\theta}{\cos\theta} \right|=0$$ so the function is differentiable in $(0,0)$ and so in $\mathbb{R}^2$.
Is my procedure correct? Is there any faster way or smart observation I can do to work this out better? Thanks in advance.
The procedure is not entirely correct because of your first inequality; by ignoring the $k^6$ term, you've created an issue in the denominator regarding division by zeros. In fact, the function isn't even continuous at the origin; consider the path $t \mapsto (t^3, t)$ in the plane, and let's evaluate the limit as $t \to 0$.
Then, for $t \neq 0$, \begin{align} f(t^3, t) &= \dfrac{t^6}{2t^6} = \dfrac{1}{2}, \end{align} so the limit as $t \to 0$ is also $\dfrac{1}{2}$. But clearly, if you take the limit along the path $t \mapsto (t,0)$ (i.e along the $x$-axis), you find the limit is $0$. Getting two different answers along different paths means the function isn't continuous.
I see several solutions regarding limits, where they go to polar coordinates to "solve" the problem. This is very dangerous, because it very often leads to subtle errors. In particular, you're assuming that $\theta$ is fixed, and that you're sending $\rho \to 0$. In other words, you're only evaluating the limit along a straight line! This is definitely not what the limit definition requires. You have the be able to approach the origin in any possible manner, and still obtain the same limit.