I'm trying to following proof of this.
$A$ which is linear mapping of complex Euclidean space into itself.
Then $A$ can be factorized by
$A=RU$ where $R$ is nonnegative self-adjoint mapping, and $U$ is unitary.
I already did when $A$ is invertible, but I'm stuck with not invertible case.
If $A$ is not invertible,
Scalar product $(AA^*x,x)=(A^*x,A^*x) \geq 0$. Thus, $AA^*$ is nonnegative.
By Theorem, $AA^*$ has unique nonnegative self-adjoint square root. Denote it $R$.
$\parallel Rx \parallel^2=(Rx,Rx)=(R^2x,x)=(AA^*x,x)=(A^*x,A^*x)=\parallel A^*x\parallel^2$
Suppose $Rx=Ry$.
Then $\parallel R(x-y)\parallel=0$. Thus, $\parallel A^*(x-y)\parallel=0$. Therefore, $A^*x=A^*y$
---Until this, I can easily follow. But here, I cannot follow.---
This shows that for any $u$ in the range of $R$, $u=Rx$, we can define $Vu$ as $A^*x$
According to upper equation, which is
$\parallel Rx \parallel^2=(Rx,Rx)=(R^2x,x)=(AA^*x,x)=(A^*x,A^*x)=\parallel A^*x\parallel^2$,
$V$ is an isometry.
---Here, I can't understand why $V$ is an isometry from that equation.
Can someone give me some help?
$V$ is an isometry because it maps $Rx$ to $A^*x$, and by your "upper equation", $\|Rx\|=\|A^*x\|$. By the way, this only gives $V$ as an isometry from $\operatorname{ran}R$ to $\operatorname{ran}A^*$. You now have to extend it to an isometry on the whole space.
You're looking at the polar decomposition: https://en.wikipedia.org/wiki/Operator_theory#Polar_decomposition
If you have access to Axler's book ("Linear Algebra Done Right"), he has a good exposition in 7.D (p.234).