If $m$ and $n$ are integers, show that $\biggl|\sqrt{3}-\dfrac{m}{n}\biggr| \ge \dfrac{1}{5n^{2}}$.
Since $\biggl|\sqrt{3}-\dfrac{m}{n}\biggr|$ is equivalent to $\biggl|\dfrac{ \sqrt{3}n-m}{n}\biggr|$
So I performed the following operation $\biggl|\dfrac{\sqrt{3}n-m}{n}\biggr|\cdot \biggl|\dfrac{\sqrt{3}n+m}{\sqrt{3}n+m}\biggr|$ to get $$\biggl|\dfrac{3n^{2}-m^{2}}{\sqrt{3}n^{2}+mn}\biggr|$$
Since $n,m \ne 0$, we have that $|3n^{2}-m^{2}| \ge 1$. Now for the denominator, we have $$ |\sqrt{3}n^{2}+mn| \le |\sqrt{3n^{2}}| + |mn| $$
Thus it follows that $$\dfrac{1}{|\sqrt{3}n^{2}+mn|} \ge \dfrac{1}{|\sqrt{3}n^{2}| + |mn|}$$
Would I have to work in cases where $m<n$, for example? Then we have $$|\sqrt{3}n^{2}| + |mn| < |\sqrt{3}n^{2}| + n^{2} < 3n^{2} + n^{2} < 5n^{2}$$ which gives us the desired result. Although, the same method doesn't work when $n >m$.
You're asking to prove, for integers $m$ and $n$ (with the assumption $n \neq 0$), that
$$\left|\sqrt{3}-\frac{m}{n}\right| \le \frac{1}{5n^2} \tag{1}\label{eq1A}$$
Note if $m = 0$, \eqref{eq1A} obviously holds. Otherwise, as this other answer states, WLOG, we may assume both $m$ and $n$ are positive since if they have opposite signs, the result is trivial, and if they are both negative, the result is the same as if they were both their absolute value equivalents instead.
As you've shown by rationalizing the numerator and stating it must be at least $1$ is that you have
$$\left|\sqrt{3}-\frac{m}{n}\right| = \left|\frac{3n^2-m^2}{\sqrt{3}n^2 + mn}\right| \ge \frac{1}{\sqrt{3}n^2 + mn} \tag{2}\label{eq2A}$$
If the denominator on the right side is $\le 5n^2$, then you get
$$\begin{equation}\begin{aligned} \sqrt{3}n^2 + mn & \le 5n^2 \\ \frac{1}{\sqrt{3}n^2 + mn} & \ge \frac{1}{5n^2} \end{aligned}\end{equation}\tag{1}\label{eq3A}$$
so combined with \eqref{eq2A}, this shows \eqref{eq1A} will be true.
Consider instead that the denominator is $\gt 5n^2$ to get
$$\begin{equation}\begin{aligned} \sqrt{3}n^2 + mn & \gt 5n^2 \\ mn & \gt (5 - \sqrt{3})n^2 \\ m & \gt (5 - \sqrt{3})n \\ \frac{m}{n} & \gt 5 - \sqrt{3} \\ -\frac{m}{n} & \lt - 5 + \sqrt{3} \\ \sqrt{3} -\frac{m}{n} & \lt - 5 + 2\sqrt{3} \lt -1.5 \\ \left|\sqrt{3}-\frac{m}{n}\right| & \gt 1.5 \gt \frac{1}{5n^2} \end{aligned}\end{equation}\tag{4}\label{eq4A}$$
As such, \eqref{eq1A} will still hold in this case as well. Since all possibilities have been covered, it proves \eqref{eq1A} is always true.