The definition given at class was:
$f(x,y)$ is differentiable at $(x_0,y_0)$ if there exists $a,b \in \mathbb{R}$ such that for $h,k \in \mathbb{R}$, if $ (h,k) \rightarrow (0,0)$, then: $$\frac{f(x_0+h,y_0+k) - f(x_0,y_0) - (ah+bk)}{\sqrt{h^2+k^2}} \rightarrow0$$
I will try to show that the fraction above doesn't converge to $0$ for any $a,b$.
Let $a,b \in \mathbb{R}\setminus(0,0)$ and $(x_0,y_0) = (0,0)$. $$ \lim_{(h,k)\rightarrow (0,0)} \frac{f(0+h,0+k) - f(0,0) - (ah+bk)}{\sqrt{h^2 + k^2}} = $$ $$ =\lim_{(h,k)\rightarrow (0,0)} \frac{\sqrt{|hk|} - ah - bk}{\sqrt{h^2+k^2}} $$ Let's check the limit: $$\lim_{h\rightarrow0} \lim_{k\rightarrow0} [\cdots] = \lim_{k\rightarrow0} 0 - 0 -\frac{bk}{|k|} = \pm b \neq 0$$ So even if the very first limit exists, it is not $0$. Now let $a,b = 0$ and $h = k = \frac{1}{n}$. Then $$ \lim_{x,y\rightarrow 0,0} \frac{\sqrt{|hk|}}{\sqrt{h^2+k^2}} = lim_{n\rightarrow \infty} \frac{\frac{1}{n}}{\frac{\sqrt{2}}{n}} = \frac{1}{\sqrt{2}} \neq 0 $$
Is this valid proof? I wasn't given answer for this problem and I prefer not to trust online calculators.
Yes it is a valid proof. Fundamentally you show that $\frac{\partial f}{\partial x}(0)=\frac{\partial f}{\partial y}(0)=0$, so if $f$ was differentiable at $0$, then its differential would be $0$. However you differentiate in another direction (actually any direction works) and show that this does not give zero, which contradicts the linearity of the differential. So $f$ cannot be differentiable at $0$.