Ok so to start, I found an expression for $\frac{|\vec{y}|^2}{|\vec{x}|^2}$ where $$|\vec{y}|^2=\vec{y}^T\cdot{\vec{y}}=(A\vec{x})^T(Ax)=\vec{x}^TA^TA\vec{x}$$ therefore: $$\frac{|\vec{y}|^2}{|\vec{x}|^2}=\frac{\vec{x}^TA^TA\vec{x}}{\vec{x}^T\vec{x}}$$
From this, $(A^TA)^T=A^TA$ so the matrix $A^TA$ is a symmetric matrix and therefore has orthogonal eigenvectors. Let these be $\vec{u_1}$, $\vec{u_2}$ and $\vec{u_3}$ corresponding to eigenvalues $\lambda_1$, $\lambda_2$, $\lambda_3$. We can then say $\vec{x}=\alpha\vec{u1}+\beta\vec{u2}+\gamma\vec{u3}$.
Using this, we can deduce that: $$\frac{|\vec{y}|^2}{|\vec{x}|^2}=\frac{\alpha^2\lambda_1+\beta^2\lambda_2+\gamma^2\lambda_3}{\alpha^2+\beta^2+\gamma^2}$$
Now assuming that $\lambda_1<\lambda_2<\lambda_3$, can we conclude that: $$\lambda_1<\frac{|\vec{y}|^2}{|\vec{x}|^2}<\lambda_3$$
Am I correct in my reasoning here? Also how do I find the value of x such that the minimum value of $\lambda_1$ is satisfied? Thanks for any help you can offer.
First of all, note that since $|y|\ge 0$, all the eigenvalues of $A^TA$ are nonnegative.
The below essentially gives you two ways to solve this: One way using Gaussian elimination, the other using the power iteration algorithm.
If $det(A^TA)=0$ (which, if you are using the power iteration algorithm, you don't need to test for separately...you will find this out when you try to invert $A^TA$), solve $A^TAx=0$ for a non-zero $x$. You can figure this out from the characteristic polynomial or the minimal polynomial of $A^TA$, among other ways. If the minimal polynomial of $A^TA$ is $m(t)=t^n+a_{n-1}t^{n-1}+...+a_1t$ (no constant term since $A^TA$ is not invertible), just apply $r(t)=m(t)/t$ to $A^TA$ and the result will be non-zero, and any non-zero column will be an $x$ with $A^TAx=0$. You can also solve $A^TAx=0$ by Gaussian elimination.
If $det(A^TA)\ne 0$, apply the power iteration algorithm to $(A^TA)^{-1}$. You can look up the power iteration algorithm on Wikipedia. It will find the eigenvector corresponding to the largest eigenvalue of $(A^TA)^{-1}$, which will also be the eigenvector corresponding to the smallest eigenvalue of $A^TA$.
Or in the $det(A^TA)\ne 0$ case, you can also use the characteristic polynomial to determine the smallest eigenvalue $\lambda$ and then apply Gaussian elimination to solve $(A^TA-\lambda I)x=0$.