Show that if $a_n = n(n \sqrt{2} - \lfloor n \sqrt{2} \rfloor )$ and if $\epsilon > 0$ then $$a_n < \frac{1+\epsilon}{2\sqrt{2}}$$ for infinitely many $n$ while $$a_n > \frac{1-\epsilon}{2\sqrt{2}}$$ for all sufficiently large $n$.
I suspect that this has to do with Pell's equation but do not know where to start nor proceed.
if we call $m = \lfloor n \sqrt 2 \rfloor \; ,$ for your first part you want $n \sqrt 2 - m$ small, but positive. Well, $$ (n \sqrt 2 - m) (n \sqrt 2 + m) = 2 n^2 - m^2 \; .$$ If positive, this is at least $1.$ In fact, there are an infinite sequence of $n$ values for which $2 n^2 - m^2 = 1$ is possible, or $2 n^2 - 1$ is a square. The smallest such $n$ are $1, 5.$ The $n$ values actually obey a Fibonacci type recurrence, $n_{j+2} = A n_{j+1} - n_j.$ Can you find $A \; ?$