This is a snapshot of Baby Rudin:
Theorem For every real $x>0$ and every integer $n>0$ there is one and only one positive real $y$ such that $y^n=x$.
This number $y$ is written $\sqrt[n]{x}$ or $x^{1/n}$.
Proof That there is at most one such $y$ is clear, since $0<y_1<y_2$ implies $y_1^n<y_2^n$.
Let $E$ be the set consisting of all positive real numbers $t$ such that $t^n<x$.
If $t=\frac{x}{1+x}$ then $0\le t<1$. Hence $t^n\le t<x$. Thus $t\in E$, and $E$ is not empty.
If $t>1+x$ then $t^n\ge t>x$, so that $t\notin E$. Thus $1+x$ is an upper bound of $E$.
Hence Theorem 1.19 implies the existence of
$$y=\sup E$$
To prove that $y^n=x$ we will show that each of the inequalities $y^n<x$ and $y^n>x$ leads to a contradiction.
The identity $b^n-a^n=(b-a)(b^{n-1}+b^{n-2}a+\cdots+a^{n-1})$ yields the inequality
$$b^n-a^n<(b-a)nb^{n-1}$$
when $0<a<b$.
Assume $y^n<x$. Choose $h$ so that $0<h<1$ and
$$h<\frac{x-y^n}{n(y+1)^{n-1}}$$
Put $a=y,b=y+h$. Then
$$(y+h)^n-y^n<hn(y+h)^{n-1}<hn(y+1)^{n-1}<x-y^n$$
Thus $(y+h)^n<x$, and $y+h\in E$. Since $y+h>y$, this contradicts the fact that $y$ is an upper bound of $E$.
Assume $y^n>x$. Put
$$k=\frac{y^n-x}{ny^{n-1}}$$
Then $0<k<y$. If $t\ge y-k$, we conclude that
$$y^n-t^n\le y^n-(y-k)^n<kny^{n-1}=y^n-x$$
Thus $t^n>x$, and $t\notin E$. If follows that $y-k$ is an upper bound of $E$.
To get the definition of $h$, I understand we need $x-y^n$ in the denominator to make the whole thing greater than $0$, but where does the denominator $n(y+1)^{n-1}$ come from?
The same question on the form of $k$. Why do we need such a denominator $ny^{n-1}$?
Rudin follows the mathematical style of writing his 'probe for the proof' on a piece of scrap paper, and upon arriving at the solution, writing up a formal and terse proof that hides the 'mathematical eyebrow sweat' that went behind it.
The $\text{1.21 Theorem}$ the OP is analyzing comes right after Rudin introduces the real numbers as a complete ordered field in Chapter 1; the solution there can only use the developed axiomatic machinery.
Let us probe this matter and work up some 'sweat'.
We have our fixed positive real number $x$ and an integer $n \gt 0$. In general the only easy thing is to find a real number $a \gt 0$ such that
$\tag 1 a^n \lt x$
Can we do better? Can we 'make $a$ larger' so that $\text{(1)}$ still holds?
Suppose $b \gt a$. How do we 'enforce' the condition that $b^n \lt x$?
$\tag 2 [ b^n \lt x ] \text{ iff } [ (b^n - a^n + a^n) \lt x ] \text{ iff } [ (b^n - a^n) \lt (x - a^n) ]$
It may take you a page or two of scrap paper to discover that the key to solving this is that
$\tag 3 b^n-a^n<(b-a)nb^{n-1}$
is always true with $0 \lt a \lt b$. So by setting up some bound $M$ and with $b = a + h$, you can write
$\tag 4 b^n \lt h M + a^n \lt hM + x$
Geometrically, Rudin knows that $E$ is an interval. The set $F$ of real positive numbers $u^n \gt x$ is also an open interval. The supremum of $E$ is equal to the infimum of $F$.
The above describes the conceptual thinking behind Rudin's proof (no calculus allowed).
The following theory gets the result in a different manner, and proposition 2 clarifies part of Rudin's argument in his proof construction.
Let $x$ be some fixed positive real number and $n \in {\mathbb Z}^+$.
Proposition 1: Let $D \subset (0, +\infty)$ be a nonempty set of real numbers and assume that for any $d \in D$, $d^n \gt x$. Then $[\text{inf(D)}]^n \ge x$.
Proof
Let $y = \text{inf(D)}$ and to arrive at a contradiction, assume that $y^n \lt x$. Choose $h$ so that $0<h<1$ and
$\quad h<\frac{x-y^n}{n(y+1)^{n-1}}$
Set $a=y$ and $b=y+h$. Then, using Rudin's (3) from the prior section,
$\quad (y+h)^n-y^n<hn(y+h)^{n-1}<hn(y+1)^{n-1}<x-y^n$
But then $(y+h)^n<x$ and so $y + h$ must be a lower bound for $D$. Yet it is greater than $y$ which is the $\text{glb}$, a contradiction. $\blacksquare$
Proposition 2: Let $D \subset (0, +\infty)$ be a nonempty set of real numbers and assume that for any $d \in D$, $d^n \lt x$. Then $[\text{sup(D)}]^n \le x$.
Proof
Let $y = \text{sup(D)}$ and to arrive at a contradiction, assume that $y^n \gt x$. Set
$\quad h=\frac{y^n-x}{ny^{n-1}}$
It is easy to see that $0 \lt h \lt y$.
Set $a=y-h$ and $b=y$. Then, using Rudin's (3) from the prior section,
$\quad y^n-(y-h)^n<hny^{n-1}= y^n-x$
But then $(y-h)^n>x$ and so $y - h$ must be an upper bound for $D$. Yet it is smaller than $y$ which is the $\text{lub}$, a contradiction. $\blacksquare$
Let $E = \{t \gt 0 \, | \, t^n \lt x \}$ and $F = \{t \gt 0 \, | \, t^n \gt x \}$.
Proposition 3: $\text{sup(E)} = \text{inf(F)}$ and if we denote this number by $\gamma$, then $\gamma^n = x$.
Proof
Let $\beta = \text{inf(F)}$. By proposition 1, $\beta^n \ge x$.
Let $\alpha = \text{sup(E)}$. By proposition 2, $\alpha^n \le x$.
Since $\beta$ is an upper bound for $E$, $\alpha \ge \beta$.
Since $\alpha$ is an lower bound for $F$, $\alpha \le \beta$.
So $\alpha = \beta$ and the proposition follows. $\blacksquare$
In a comment the OP had a question on how to derive the inequality. To be complete, even if it is no longer necessary, consider this:
The identity $b^n-a^n=(b-a)(b^{n-1}+b^{n-2}a+\cdots+a^{n-1})$ yields the inequality
$b^n-a^n<(b-a)nb^{n-1}$
when $0<a<b$.
We have $n$ terms in $b^{n-1}+b^{n-2}a+\cdots+b^0 a^{n-1}$, but since $a \lt b$,
$|b^{n-1}a^0+b^{n-2}a^1+\cdots+b^0 a^{n-1}| \lt |b^{n-1}a^0|+|b^{n-2}a^1|+\cdots+|b^0 a^{n-1}| \lt$
$\quad |b^{n-1}|+|b^{n-1}|+\cdots+|b^{n-1}| \lt n b^{n-1}$
In his question, the OP ask where does the denominator come from in
$\quad h<\frac{x-y^n}{n(y+1)^{n-1}}$
This is the same thing as working with an $[\varepsilon , \delta]$ proof in calculus. After the 'probing', we set $h$ (like we do with $\delta$) to what works and shows that we can control the quantities. In essence, we are using the fact that the function $f(x) = x^n$ is continuous, but the book still hasn't discussed that concept.