To check the existence of $f_{xy}(0,0)$ and $f_{yx}(0,0)$ for a given function $f(x,y)$
$$f(x,y) =\begin{cases} x^2\tan^{-1}\frac{y}{x} + y^2\tan^{-1}\frac{x}{y} , &(x,y) \neq (0,0)\\ \\0, & (x,y) = (0,0)\end{cases}$$
Now $$f_x(0,0) = \lim_{h \to 0} \frac{f(h,0) - f(0,0)}{h} $$
$f(h,0)$ clearly does not exist so $f_x(0,0)$ does not exist and hence $f_{xy}(0,0)$ does not exist?
Is this correct? The solution given finds $f_x(0,0)$ as $0$ ( It is pretty clear how they got there but isn't it wrong since $\tan^{-1}\frac{x}{y}$ is not defined at $y = 0$
In the comments I said that it doesn't make sense to talk about differentiability at the point which is not in the domain of the function, which is true, but it's not applicable here since $(0,0)$ is in the domain.
The correct way to look at this is that there is no open subset of $\mathbb R^2$ containing $(0,0)$ that lies in the domain of the function, which you observed by noting that $f(h,0)$ doesn't exist for $h\neq 0$.
The natural thing to do is to first extend $f$ continuously to whole of $\mathbb R^2$. Let's take a look at $x^2\tan^{-1}\frac{y}{x}$. It will converge to $0$ when $x\to 0$ by the squeeze theorem because $\tan^{-1}$ is bounded, regardless of $y$. Similarly for the other summand. This tells us that $f$ can be extended continuously to whole $\mathbb R^2$ if $f(x,y) = 0$ whenever $xy = 0$, as Anne Bauval pointed out in the comments. In that case, the partial derivatives indeed become $0$ at $(0,0)$.
However, there is a bit more work to calculate second partial derivatives:
$$f_{xy}(0,0) = \lim_{h\to 0}\frac{f_x(0,h)-f_x(0,0)}{h} = \lim_{h\to 0}\frac{f_x(0,h)}{h},$$
and
$$f_x(0,h) = \lim_{t\to 0}\frac{f(t,h)-f(0,h)}{t} = \lim_{t\to 0}\frac{f(t,h)}{t} = \lim_{t\to 0}\left(t\tan^{-1}\frac ht+h^2\frac{\tan^{-1}\frac th}t\right) = h$$
so $f_{xy}(0,0) = 1$. The other second partial derivative follows from symmetry.