Problem says:
Let $x^2+y^2=100$, where $x,y>0$. For which ratio of $x$ to $y$, the value of $x^2y$ will be maximum?
I know these possible tools:
AM-GM inequality
Calculus tools
Here, I want to escape from all of the tools I mentioned above.
I will try to explain my attempts in the simplest sentences. (my english is not enough, unfortunately). I will not prove any strong theorem and also I'm not sure what I'm doing exactly matches the math, rigorously.
Solution I made:
First, it is not necessary to make these substitutions. I'm just doing this to work with smaller numbers.
Let, $x=10m, ~y=10n$,
where $0<m<1,~ 0<n<1$, then we have
$$ x^2+y^2=100 \iff m^2+n^2=1$$
$$x^2y=1000m^2n$$
This means,
$$\max\left\{x^2y\right\}=10^3\max\left\{m^2n\right\}$$
$$m^2n=n(1-n^2)=n-n^3$$
Then suppose that,
$$\begin{align}\max\left\{n-n^3 \mid 0<n<1\right\}&=a, a>0&\end{align}$$
This implies
$$n-n^3-a≤0,~ \forall n\in\mathbb (0,1)$$
$$n^3-n+a≥0,~\forall n\in\mathbb (0,1)$$
Then, we observe that
$$\begin{align}n^3-n+a≥0, \forall n\in (0,1) ~ \text{and} ~ \forall n≥1\end{align}$$
This follows
$$ n^3-n+a≥0, ~ \forall n>0.$$
Using the last conclusion, I assume that there exist $u,v>0$, such that
$$n^3-n+a=(n-u)^2(n+v)≥0.$$
If $n>0$, then the equality occurs, if and only if
$$n=u>0$$
Based on these, we have:
$$\begin{align}n^3-n+a= (n-u)^2(n+v)≥0 \end{align}$$
$$\begin{align}n^3-n+a = & n^3 - n^2(2u-v)+ n(u^2 - 2 u v ) + u^2v & \end{align}$$
$$\begin{align} \begin{cases} 2u-v=0 \\ u^2-2uv=-1 \\u^2v=a \\u,v>0 \end{cases} &\implies \begin{cases} v=2u \\ u^2-4u^2=-1 \\ 2u^3=a \\ u,v>0 \end{cases}\\ &\implies \begin{cases} u=\frac{\sqrt 3}{3} \\ v=\frac{2\sqrt 3}{3}\\ a=2\left(\frac{\sqrt 3}{3} \right)^3=\frac{2\sqrt 3}{9} \end{cases} \end{align}$$
$$\begin{align}n^3-n+\frac{2\sqrt 3}{9} &=\left(n-\frac{\sqrt 3}{3} \right)^2\left(n+\frac{2\sqrt 3}{3}\right)≥0.&\end{align}$$
As a result, we deduce that
$$\begin{align}n-n^3-\frac{2\sqrt 3}{9} &=-\left(n-\frac{\sqrt 3}{3} \right)^2\left(n+\frac{2\sqrt 3}{3}\right)≤0, &\forall n\in (0,1).&\end{align}$$
$$\begin{align}\max\left\{n-n^3 \mid 0<n<1\right\}&=\frac{2\sqrt 3}{9}, ~ \text{at }~ n=\frac{\sqrt 3}{3}&\end{align}$$
Finally, we obtain
$$m=\sqrt{1-n^2}=\sqrt{1-\frac 13}=\frac{\sqrt 6}{3}$$
$$\frac xy=\frac mn=\sqrt 2.$$
Question:
- How much of the things I've done here are correct?

Your approach is good and the easiest way without calculus.
An Approach that Exposes the Core Ideas
The following approach is a slight modification of yours that simplifies the algebra using Vieta's formulas.
If $$ x^2+y^2=100\tag1 $$ then $$ \begin{align} x^2y &=100\cos^2(\theta)\cdot10\sin(\theta)\tag{2a}\\ &=1000\left(\sin(\theta)-\sin^3(\theta)\right)\tag{2b} \end{align} $$ So we wish to maximize $\sin(\theta)-\sin^3(\theta)$ for $0\lt\theta\lt\frac\pi2$ (since $x,y\gt0$). Suppose the maximum is $m$. Then your assumption/lemma says that $s-s^3-m=0$ has a double root. That is, $$ s^3-s+m=(s-r)^2(s-q)\tag3 $$ Vieta's formulas say that for a monic cubic polynomial, $$ \begin{align} r+r+q&=0\tag{4a}\\ r^2+rq+rq&=-1\tag{4b}\\ r^2q&=-m\tag{4c} \end{align} $$ Explanation:
$\text{(4a)}$: the sum of the roots is the negative of the coefficient of $s^2$
$\text{(4b)}$: the sum of the pairwise products of the roots is the coefficient of $s$
$\text{(4c)}$: the product of the roots is the negative of the constant coefficient
Now things just fall into place:
$\text{(4a)}$ says that $q=-2r$
$\text{(4b)}$ says that $-3r^2=-1$, that is $r=\frac1{\sqrt3}$
$\text{(4c)}$ says that $-2r^3=-m$, that is $m=\frac2{3\sqrt3}$
Thus, the maximum of $s-s^3$ is $\frac2{3\sqrt3}$ which happens at $s=\frac1{\sqrt3}$, which is a value that $\sin(\theta)$ attains for $0\lt\theta\lt\frac\pi2$.
Thus, the maximum of $x^2y=1000\cdot\frac2{3\sqrt3}=\frac{2000}{3\sqrt3}$.