Theorem 161: For each ζ, the equation ξξ = ζ has exactly one solution.
In part II of the proof the author considers the set of all rational numbers for which X~X~ < ζ. And claims that it constitutes a cut. Where X is a rational number (class of equivalent fractions) . And X~ is the set of all rational numbers < X. Which is a cut by theorem 150. Rational cuts have been renamed rational numbers at this point in the text due to the results of chapter III section 5.
To prove property 2 of cuts it suffices to show:
With every number it contains, the set also contains all numbers smaller than that number.
The author proves property 2 in the following way:
If X~X~ < ζ, Y~ < X~ then Y~Y~ < X~X~ < ζ
However if we consider a ζ of 10~ say. An X~ of 3~ say. And a Z~ of 2~ < X~X~. Then by theorem 162 (√2 is irrational) our Z~ cannot be of the form R~R~ = Z~. Therefore X~X~ < ζ does not constitute a cut, by counterexample.
You define the notation X~ to imply that X is a rational number and then X~ shall mean the set of rationals less than X. Now you take $\zeta$ to be 10~, so that the squareroot of 10 is being defined. You are correct in saying that (3~)(3~) is less than 10~. [In fact one easily shows that (3~)(3~)=9~.]
Now property 2 of cut, applied here with its X~ taken to be your 3~, only asserts that provided Y~ is less than 3~, it should also be the case that (Y~)(Y~) is less than 10~.
Provided the rationals denoted X,Y,Z (and A,B for the following "lemma") here are positive, it is an easy matter to show the "lemma":
A~ < B~ if and only if (A~)(A~) < (B~)(B~).
From this we can get from the assumption Y~ < 3~ to the intermediate conclusion (Y~)(Y~) < (3~)(3~), and thence to the desired (Y~)(Y~) < 10~ by transitivity. All of this has nothing to do with Z~ being 2~, as that is just a particular Y~ less than 3~. In fact nowhere in this argument you present are any irrational cuts directly used, since each is explicitly of the form X~, which you define as a rational cut at the rational number X.
ADDED: I might see what your confusion is: When one takes the specific Y~ which happens to be less than some product X~X~, in turn less than $\zeta$, notice that no attempt is made in the statement to express that Y~ itself as a product of two other rational cuts. Instead, the only further use of Y~ made in the statement of cut property 2 is to form the product of Y~ with itself. Since given any rational cut Y~ there is no problem squaring it, there is no problem. In fact (provided Y > 0) you just have Y~Y~=(Y^2)~.