My approach is based on this method: we want to find $U$, $\Sigma$, $V$ so that $A = U \Sigma V$.
Then $A^T A = V^T \Sigma U^T U \Sigma V$. Since $U$ is an orthogonal matrix, this equals $V^T \Sigma^2 V$, therefore $V$ can be computed as the transpose of the $ATA$'s eigenvector matrix.
Similarly, $AA^T = U \Sigma U^T$.
The only assumption we made here was that $U$ and $V$ are orthogonal, but that doesn't seem to be enough for me. Let's take an example:
$$A = \begin{bmatrix}2&5\\3&6\\4&7\end{bmatrix}$$
The calculated SVD components are:
$$U = \begin{bmatrix}-0.46&0.79&0.41\\-0.57&0.09&-0.82\\-0.68&-0.6&0.41\end{bmatrix}$$
$$\Sigma = \begin{bmatrix}11.77&0&0\\0&0.62&0\\0&0&0\end{bmatrix}$$
$$V = \begin{bmatrix}-0.45&-0.89\\-0.89&0.45\end{bmatrix}$$
This works, but here comes my question is: $AA^T$ second eigenvector could be also
$$\begin{bmatrix}-0.79&-0.09&0.6\end{bmatrix}$$
This doesn't change $U$'s orthogonality,the factorization is not good anymore. Why is that? I know that the above method is not a precise proof (because of the size of the matrices), but I don't think think this causes the problem.
So the question would be what other assumptions should be made on $U, V$ to get the proper factorization?
As far as I understood, your question and observation is essentially: Is the singular value decomposition unique? And the short answer is no. The singular vectors are only unique up to a multiplication of (-1). This can be seen by the construction of the singular vectors for a matrix $A$. The right-singular vectors can be constructed by:
$$v_k = \arg\max_{\|v\|_2 = 1, \langle v, v_i \rangle = 0, \text{ for }i<k} \|Av\|_2 $$
Here one can see, that $v_k$ and $-v_k$ are both solutions of this. The corresponding singular value is given by:
$$ \sigma_k = \|Av_k\|_2$$
which is unique, and finally, the left-singular values can be calculated by:
$$u_k = \frac{1}{\sigma_k} Av_k $$
From this we can see, that even though we can choose between $v_k$ and $-v_k$, the left-singular vectors change accordingly, such that the decomposition is still valid.
I could not find a source with a thorough explanation of this construction method. The script of my professor is unfortunately not publicly available. However, this medium post does give a good intuition.