Suppose I have a rational number $a$ and $a^2 < 2$. Can I find another rational number $B$ such that $a^2<B^2<2$?
Based on the answer to this question, I thought of doing the following:
$$ a^2 < 2 \implies a < \frac{2}{a}\\ \text{Let}\hspace{1cm} B=\frac{a+\frac{2}{a}}{2}=\frac{a^2+2}{2a} $$
$B$ is greater than $a$ because:
$$ \begin{array} {aa} B>a & \implies \frac{a^2+2}{2a}>a \\ & \implies a^2 + 2 > 2a^2 \\ & \implies 2 > a^2 \\ & \implies a^2 < 2 \end{array}$$
If $B^2$ is less than $2$, then $B^2-2<0$, but:
$$\begin{array} {aa} B^2-2 < 0 & \implies \left( \frac{a^2+2}{2a} \right)^2 - 2 < 0 \\ & \implies \frac{a^4+4a^2+4}{4a} - \frac{8a^2}{4a^2} < 0 \\ & \implies \frac{(a^2-2)^2}{(2a)^2} < 0 \end{array}$$
Which is a contradiction since the left hand side of the inequality will be positive for all values of $a$.
But I think we must be able to find such a $B$ since based on my understanding of this answer, we can find a another rational number whose distance from $a$ is less than the distance between $a$ and $\sqrt{2}$
Therefore, I have 2 questions to ask:
Why does this approach work in the case of $a^2>2$ but not when $a^2<2$?
How should I approach these kind of questions since it seems that there are a few ways to construct a $B$ that satisfies a given set of restrictions? For example, see here (the proof is immediately before the section "13. The Completeness Axiom".
The substitution $$a \leftarrow \frac{a^2+2}{2a} $$ is exactly what you get from Newton's method for the function $f(a)=a^2-2$. This will result in iterations that are greater than $\sqrt{2}$ since $f$ is an increasing convex function on $(0, \infty)$. I will sketch two alternatives, both based on Newton's method, to get iterations less than $\sqrt{2}$.
First idea. Use some other function than $f(a) = a^2-2$ with Newton's method. For example, if $0 \leq p < \sqrt{2}$ then
$$ \frac{1}{\sqrt{2}+p} = \frac{\sqrt{2}-p}{2-p^2} $$
and so $\sqrt{2}$ is a root of the function $$f(a) = \frac{2-p^2}{a+p}-a+p = \frac{2-a^2}{a+p}.$$
This function is convex and decreasing on $(0, \infty)$. Applying Newton's method to this function results in the iteration
$$ a \leftarrow \frac{p a^2+4 a+2 p}{a^2+2 p a+2} $$
which produces an increasing sequence that converges to $\sqrt{2}$ from below as required. For example for $p=0$ you get
$$ a \leftarrow \frac{4a}{a^2 + 2} $$
which is exactly the harmonic mean of $a$ and $\frac{2}{a}$ as suggested by haruspex. For $p=1$ and $p=\frac{7}{5}$ you get
$$ a \leftarrow \frac{a^2+4a+2}{a^2+2a+2} \textrm{ and } a \leftarrow \frac{7 a^2+20 a+14}{5 a^2+14 a+10} $$
respectively, to give just two other examples. Substituting $a=\frac{7}{5}$ in the latter results in $\frac{1393}{985}$ which is less than $4\times 10^{-7}$ below $\sqrt{2}$.
Second idea. For this idea I will assume that $a\geq 1$. As you already found out we have $$a < \sqrt{2} < \frac{a^2+2}{2a}.$$
Therefore we could try to take some weighted average of $a$ and $\frac{a^2+2}{2a}$ that ends up below $\sqrt{2}$. So we're looking for some factor $\lambda \in (0,1)$ such that
$$ \lambda a + (1-\lambda)\frac{a^2+2}{2a} < \sqrt{2} $$
for all $a \in [1, \sqrt{2})$. It is not difficult to show that this is the case exactly if $\lambda \in (3 - 2\sqrt{2}, 1)$. We can take $\lambda = \frac{1}{5}$ or $\lambda = \frac{5}{29}$ for example to get the iterations
$$ a \leftarrow \frac{3 a^2+4}{5a} \textrm{ and } a \leftarrow \frac{17 a^2 + 24}{29a} $$
respectively.