How does one proceed in solving this inequality?
X is of full column rank
$$\left\|\left(\boldsymbol{X}^{T} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{T} \boldsymbol{v}\right\|_{2} \leq \frac{1}{\sigma_{\min }(\boldsymbol{X})}\|\boldsymbol{v}\|_{2}$$
Let $X=U\Sigma V^\top$ be the SVD of $X$. Note that $\Sigma$ has the same shape as $X$ and also has full column rank.
Then, $$(X^\top X)^{-1} X^\top = V(\Sigma^\top \Sigma)^{-1} \Sigma^\top U.$$ The operator norm of this matrix is the same as the operator norm of $(\Sigma^\top \Sigma)^{-1} \Sigma^\top$. From here you can explicitly do the matrix multiplication to see that this matrix is rectangular diagonal with diagonal entries $1/\sigma_i$.