Related the question Min-Max Principle $\lambda_n = \inf_{X \in \Phi_n(V)} \{ \sup_{u \in X} \rho(u) \}$ - Explanations, I tried to complete the details of mickep's answer, but I have a little problem. Here is what I've done so far :
Let us consider the matrix $$ A= \begin{bmatrix} 4 & -2\\ -2 & 7 \end{bmatrix}. $$ We know that the eigenvalues are $\lambda_1=3$ and $\lambda_2=8$ with the unit eigenvectors $u_1=\displaystyle\frac{1}{\sqrt{5}}(2,1)$ and $u_2=\displaystyle\frac{1}{\sqrt{5}(-1,2)}(-1,2)$.
In this problem, we want to consider the Rayleigh quotient with $X$ a $1$-dimensional linear subspace.
Each vector is a basis vector of such subspace $X$, and the Rayleigh quotient is constant in each of these subspaces. By definition, we got $$\lambda_1=\min_X \{\max_x \{R_A(x) : x \in X, x \not = 0 \text{ and } \dim X = 1 \}\} \\ = \min_X \{R_A(x) : x \in X, x \not = 0 \text{ and } \dim X = 1 \}. $$
So it suffices to work on a unit vector in each subspace, and we just have to find the subspace (direction) in which the Rayleigh quotient is minimal.
$$A\begin{pmatrix}x\\y\end{pmatrix} = 4 x^2-4xy+y^2 \implies \left<A\begin{pmatrix}x\\y\end{pmatrix},\begin{pmatrix}x\\y\end{pmatrix}\right> = \begin{pmatrix}x\\y\end{pmatrix}^T A \begin{pmatrix}x\\y\end{pmatrix}=Q.$$
Based on the foregoing, we could apply the Principle Axis Theorem (see here); therefore, $Q=y^T D y = \lambda_1 y_1^2+\lambda_2 y_2^2 (\text{suppose} =1) \iff \displaystyle\frac{y_1^2}{(1/\sqrt{3})^2}+\frac{y_2^2}{(1/2\sqrt{2})^2}=1$.
Then, the minimum is attained at $y_{\min}=(0, \displaystyle\frac{1}{\sqrt{8}})$ is the $y_1 y_2$- plane.
This minimum becomes $x_{\min}=P y_{\min}= \displaystyle\frac{1}{\sqrt{40}}\begin{pmatrix}-1\\2\end{pmatrix}$,
where $P= \displaystyle\frac{1}{\sqrt{5}} \begin{bmatrix} 2 & -1\\ 1 & 2 \end{bmatrix}$, is in the $xy$-plane.
From there, if I apply the Rayleigh quotient to this vector, I should obtain $\lambda_1$, but instead, I obtain $\lambda_2.$
I really don't know where I made my mistake. Is there anyone who could tell me where my mistake is and why?
Thanks!
P.S. If the question is unclear, please let me know.
You seem to be getting twisted in a knot. The actual problem is, as I'm sure you realize, pretty simple: you are trying to minimize $x^\top A x$ subject to the constraint $x^\top x = 1$. For that you can use any of several techniques, such as Lagrange multipliers.
You seem to be doing fine for a while: you've written $A = P^\top D P$ where the diagonal matrix $D$ has diagonal entries coming from the eigenvalues and the columns of $P$ come from the corresponding eigenvectors, assumed to be normalized so that $P$ becomes an orthogonal matrix. So in principle, putting $y = P x$ (here $x = (x_1, x_2)^\top, y = (y_1, y_2)^\top$), the problem of minimizing
$$x^\top A x = x^\top P^\top D P x = (P x)^\top D P x = y^\top D y$$
is transformed into the problem of minimizing $y^\top D y = 3y_1^2 + 8y_2^2$, subject to the constraint $x^\top x = 1$ as before: in $y$-terms, this constraint is the same as $y^\top y = 1$, since the orthogonal transformation $P$ preserves the inner product and hence maps the unit circle to the unit circle. So, we are trying to minimize $3y_1^2 + 8y_2^2$ subject to the constraint $y_1^2 + y_2^2 = 1$. Clearly this is minimized where $y_2 = 0$, whereupon $y_1 = 1$ or $-1$, and then $x = P^{-1} y = P^\top y$ gives what you want.
But look what you've done: you've flipped the problem, where you instead posit a constraint $3y_1^2 + 8 y_2^2 = 1$ and then (apparently) seek the minimum length vector $(y_1, y_2)$, i.e., the vector $(y_1, y_2)$ where $y_1^2 + y_2^2$ is minimized, subject to that constraint. That's exactly backwards! You've gone on to solve
instead of the actual problem, which is
I think that is the source of the confusion. It would be possible to do it the way you are (and visually the first problem may be easier to grok, where you draw an ellipse and look for the shortest vector), as long as you realize what you're doing geometrically, where inverting the relationship between what is being minimized and what is held constant will invert (or perhaps reciprocate is the better word) the order of the solutions.