I have a function given by $$f(x,y)=\frac{\sinh^2(\omega_-)}{\sinh(\omega_{in})\sinh(\omega_{ou})}$$ where $$\omega_{in}=\sqrt{x^2+\alpha y^2}$$ $$\omega_{ou}=\sqrt{x^2+\beta y^2}$$ $$\omega_-=\frac{1}{2}(\omega_{ou}-\omega_{in})$$ and $\beta>\alpha>0$.
When I plot $f(x,y)$ in Mathematica I get this I have used $\alpha=1$ and $\beta=3$ here.
It seems to have some non-differentiable (cusp/kink) behaviour around the origin but the function consists of simple exponential function whose argument is of form $\sqrt{x^2+y^2}$. What is going on here?
Just use the Taylor series for a multivariable function. Around the origin, that is for $(x, y)\to (0, 0)$ we have, term by term:
$$\sinh^2\left(\frac{1}{2}\left(\sqrt{x^2 + \alpha y^2} - \sqrt{x^2 + \beta y^2}\right)\right) \approx \frac{1}{4} \left(\sqrt{a y^2+x^2}-\sqrt{b y^2+x^2}\right)^2$$
$$\sinh\left(\sqrt{x^2 + k y^2}\right) \approx \frac{1}{6} \left(k y^2+x^2\right)^{3/2}+\sqrt{k y^2+x^2}$$
Where $k$ can be your alpha or beta.
Hence
$$f(x, y) \approx \frac{\frac{1}{4} \left(\sqrt{a y^2+x^2}-\sqrt{b y^2+x^2}\right)^2}{\left(\frac{1}{6} \left(\alpha y^2+x^2\right)^{3/2}+\sqrt{\alpha y^2+x^2}\right)\left(\frac{1}{6} \left(\alpha y^2+x^2\right)^{3/2}+\sqrt{\alpha y^2+x^2}\right)}$$
You can arrange this a bit to get
$$f(x, y) \approx \frac{9 \left(\sqrt{a y^2+x^2}-\sqrt{b y^2+x^2}\right)^2}{\sqrt{\alpha y^2+x^2} \left(\alpha y^2+x^2+6\right) \sqrt{\beta y^2+x^2} \left(\beta y^2+x^2+6\right)}$$
If you now take the limit you get
$$\lim_{(x, y) \to (0, 0)} f(x, y) = -\frac{\left(\sqrt{a}-\sqrt{b}\right)^2}{4 \sqrt{a} \sqrt{b}}$$
Assuming $(a, b) > 0$ this becomes
$$\dfrac{1}{2} - \frac{a+b}{4\sqrt{ab}}$$
So you can see that it's really well defined in the origin.
Notice that you can compute the limit of your initial function directly, without approximations, and you will obtain the same result.