So I've been working on Rudin's 1.1.3 for a while and I think I'm close to a figuring it out.
In the set $A$ there are positive rationals $p$ such that $p^2<2$ and we can find a rational $q$ in $A$ such that $p<q$. So we can also say $q^2<2$.
I want to say that $q=p+x$ where $x$ is very small so that I can say $p^2<q^2$ because $q^2=(p+x)^2=p^2+2px+x^2$ and that's bigger than just $p^2$
Now $p^2<q^2<2$ and I can move things around so that I get $0<q<p+(2-p^2)/(q+p)$ which I can then rearrange into what I have below.
So, unless I'm doing things wrong, I can get $p<q<p-(p^2-2)/(p+q)$
The latter part of that relation is somewhat close to what Rudin has in 1.1.3 but I still have inequalities and my denominator still has a $q$.
=Update=
So reading a helpful reply, I've simplified by attempts:
We know that $q^2<2$ and if $q=p+x$ then we have $(p+x)^2<2$. We can then the $x$ on one side and get this; $x(2p+x)<2-p^2$
But still not sure completely clear what the next methodological step is.
=Update 2=
For some reason I still can't make the connection from $2p+x = p + p +x$ and the $p +2 $. I understand that the latter expression is probably chosen due to elegance in how it will nicely simplify eventually.
For this last stage, I understand that
$0<p^2<q^2=(p+x)^2<2$ and because $(p+x)^2<2$ then either $p$ or $x$ is positive and less than $sqrt(2)$. I suppose we can think of it as $p+x<sqrt(2)$ even though we shouldn't be using sqrt(2). Like, sure, $p+x<sqrt(2)<2$ but then how do we get rid of that inequality when we're replacing that $p+x$ with $2$.
I'm pretty sure I understand the theory of what Rudin's doing and your replies, but the actual computation with the inequalities is messing me up.
I don't think it's necessarily easy to go from what you're doing to what he does.
You are basically looking for some small, positive number $x$ that satisfies the inequality $$x(2p + x) < 2 - p^2.$$
At this point, there are many ways to pick $x$. For example, you could pick $x$ to be any number that is both less than $p$ and less than $(2-p^2)/3p$.
But if you want to get Rudin's result, you first impose the condition $x < 2 - p$. Now it is sufficient for the inequality $$x(p+2) \leq 2 - p^2$$ to be satisfied.
So it's enough to take $$x = \frac{2-p^2}{p + 2}$$ and to check that this value of $x$ satisfies $x < 2-p$. This is straightforward to check.
What I think Rudin is really doing is this. Draw the parabola $y = x^2 - 2$. You already know a point $(p,p^2 - 2)$ on the parabola, and you want to find a better approximation than $p$ to $\sqrt{2}$, the new approximation being on the same side of $\sqrt{2}$ as $p$ is.
To do this you draw a line of positive (rational) slope $m$ through the point $(p,p^2 - 2)$ you know, and take the intersection $(q,0)$ with the $x$-axis. The fact that $m > 0$ guarantees that $q$ and $\sqrt{2}$ are on the same side of $p$. To be certain that you don't overshoot $\sqrt{2}$, all you need to do is make sure that the slope $m$ of the line is larger than that of the secant line from $(p,p^2 - 2)$ to $(\sqrt{2},0)$, which is $p + \sqrt{2}$. For convenience, Rudin takes $m = p + 2$, but he could have taken $p + 3/2$ or a lot of other things.
Addendum
When you reach $x(2p + x) < 2 - p^2$, you have a lot of freedom in how to pick $x$. In this problem, all we need to do is find one positive value of $x$ that works. The condition is equivalent to $$x < \frac{2-p^2}{2p + x}.$$ Because $x$ appears on the right side, it wouldn't make sense to simply say "Pick $x$ to be less than $(2-p^2)/(2p + x)$." Nonetheless, since $x$ can be picked as small as we like (clearly, if $(p + x)^2 < 2$, this will remain true if $x$ is replaced with a smaller positive number), we can simplify things by assuming that $x$ is less than some convenient value. In particular, $2p + x$ will be much simpler if we replace $x$ with $p$, since $x$ will no longer appear in the expression.
Consider the following chain of inequalities, which may or may not be true for a given value of $x$. $$x(2p + x) < x(3p) \leq 2 - p^2.$$ The first inequality will be true if $x < p$. The second will be true if $x \leq (2-p^2)/3p$. So any choice of $x$ that is less than the minimum of $p$ and $(2 - p^2)/3p$ will necessarily be a solution of the original inequality.
Addendum 2
In response to a comment, I will add details about the part of the answer that explains Rudin's expression for $q$.
This is very much like what I explained above for $3p$, but with $p + 2$ instead. Essentially, in order for the inequality $x(2p + x) < 2 - p^2$ to hold, it is sufficient for the following two inequalities to hold: $$x(2p + x) < x(p + 2) \leq 2 - p^2.$$ I have not written $p + 2$ here for any reason other than that it's the particular expression Rudin chose. Other expressions would also work, including $p + 3/2$, or more generally $p + a$ where $a$ is any rational number larger than $\sqrt{2}$.
When does the double inequality hold? The first part holds when $2p + x < p + 2$ or, equivalently, when $x < 2 - p$. The second part holds when $x \leq (2 - p^2)/(p + 2)$. As I mentioned in my answer, the second inequality actually implies the first, because we certainly have $(2-p^2)/(p + 2) < 2 - p$. Therefore it is enough to take $x = (2 - p^2)/(p + 2)$, or any smaller rational number.