Let $f(x,y)=\frac{yx^6+y^3+x^3y}{x^6+y^2}$ outside $(0,0)$ and $f(0,0)=0$. Show that $f$ is not differentiable at the origin.
I calculated the directional derivative at the origin in the direction of a vector $u=(h,k)$ and obtained that it equals $k$. Usually in such problems I assume that $f$ is differentiable at the origin, and then this equality must hold: $$\partial_uf(0,0)=hf_x(0,0)+kf_y(0,0),$$ and then arrive at a contradiction. In our situation however this holds because $f_x(0,0)=0,f_y(0,0)=1$.
I know that $f$ isn't continuous at $0$ (consider $y=x^3$) and hence $f$ isn't differentiable at zero, but I was wondering how can I show that it's not differentiable by not appealing to discontinuity?
If $f$ was differentiable at $(0,0)$ then, since $\frac{\partial f}{\partial x}(0,0)=0$ and $\frac{\partial f}{\partial y}(0,0)=1$, $f'(0,0)$ would have to be the linear map $L\colon\mathbb{R}^2\longrightarrow\mathbb R$ defined by $L(x,y)=y$. So, we would have$$\lim_{(x,y)\to(0,0)}\frac{|f(x,y)-y|}{\|(x,y)\|}=0.$$But this is the same thing as asserting that$$\lim_{(x,y)\to(0,0)}\frac{|x^3y|}{\sqrt{x^2+y^2}(x^6+y^2)}=0.$$This is not true. As you wrote, just take $y=x^3$.