Let $A \in \mathbb{R}^{m \times n}$ be a matrix with full rank and $m \le n$. How can we solve the problem $$ \mathop{\arg\max}\limits_{\substack{v \in \mathbb{R}^m \\ \| v \| = 1}} v^T A A^T v $$ with singular value decomposition (instead of PCA)?
Hint: For all orthogonal matrices $Q$ have $$\mathop{\arg\max}\limits_{\substack{v \in \mathbb{R}^m \\ \| v \| = 1}} f(Qv) = Q^T \mathop{\arg\max}\limits_{\substack{w \in \mathbb{R}^m \\ \| w \| = 1}} f(w). \tag{$\star$}$$
My idea (now with help from the comments and answers) Let $A = U \Sigma V^T$ bei the SVD of $A$. Then we have \begin{align*} M :=\mathop{\arg\max}\limits_{\substack{v \in \mathbb{R}^m \\ \| v \| = 1}} v^T A A^T v & = \mathop{\arg\max}\limits_{\substack{v \in \mathbb{R}^m \\ \| v \| = 1}} v^T U \Sigma V^T V \Sigma^T U^T v = \mathop{\arg\max}\limits_{\substack{v \in \mathbb{R}^m \\ \| v \| = 1}} (U^T v)^T \Sigma^2 (U^T v) \\ & \overset{(\star)}{=} U \cdot \mathop{\arg\max}\limits_{\substack{w := U^T v \in \mathbb{R}^m \\ \| w \| = 1}} w^T \Sigma^2 w = U \cdot \mathop{\arg\max}\limits_{\substack{w \in \mathbb{R}^m \\ \| w \| = 1}} \sum_{k = 1}^{m} (\sigma_k w_k)^2 \end{align*} Since the singular values $(\sigma_k)_{k \in \{1, \ldots, m\}}$ are all positive and $\sigma_1 \ge \sigma_2 \ge \ldots \ge \sigma_m > 0$, the argmax is attained at $w = (1, \underbrace{0, \ldots, 0}_{(m - 1)\text{-times}})^T$, so $M = U e_1$, which is the first left singular vector.
Edit: Second Idea Finding the argmax corresponds to finding an point of the function $f(v) = v^T A A^T v = \| v^T A \|^2$ where $f'(v) = 0$. Can I utilise that?
If you decide to change variables to $w=U^T v$, then you need to multiply by $U$ on the outside, in order to convert the argmax over $w$ into the argmax over $v$. Once you've done that, it should be clear that since $\sigma_1$ is the biggest, one argmax is obtained by setting $w_1=1$ and all other $w_k=0$. Thus you get $Ue_1$ which is to say the first left singular vector. (Of course $-Ue_1$ is another argmax, and there can be more of them if $\sigma_2=\sigma_1$.)
But in my view of the SVD, it is more intuitive to start from this property from scratch. That is, we define the right singular vectors $v_k$ to be some argmax of $\| Av \|$ over unit vectors orthogonal to $v_1,\dots,v_{k-1}$. We define the left singular vectors $u_k$ to be some argmax of $\| A^Tu \|$ over unit vectors orthogonal to $u_1,\dots,u_{k-1}$. Then it is a theorem that $Av_k=\sigma_k u_k$ and $A^T u_k=\sigma_k v_k$ (in particular, $A v_i$ is parallel to $u_i$ and $A^T u_i$ is parallel to $v_i$). You can then stitch these relations together into the usual $A=U \Sigma V^T$ form.