Suppose I want to find an approximation to the smallest eigenvalue of $y''=\lambda y$, with $y(0)=y(1)=0$.
One way to do it is to write the ansatz $y=x(1-x)+ax^2(1-x)^2$, compute the Rayleigh quotient, $$R[y]=\frac{\int_0^1 yy''dx}{\int_0^1 y^2dx}\propto\frac {2\,{a}^{2}+14\,a+35}{{a}^{2}+9\,a+21},$$ and compute its critical point to get the value $\lambda\approx -9.8697$, pretty close to the exact value. Fine.
I thought another way would be to represent the second derivative as a matrix acting on the space spanned by the normalized functions $f_1=\sqrt{30}x(1-x)$ and $f_2=\sqrt{630}x^2(1-x)^2$, namely $$M_{ij}=\int_0^1 f_i f_j'' dx.$$ This gives $$M=\left( \begin {array}{cc} -10&-2\,\sqrt {21} \\ -2\sqrt{21}&-12\end {array} \right),$$ whose eigenvalues are $-1.78$ and $-20.22$, completely off.
I expected the diagonalization would be equivalent to minimizing the Rayleigh quotient. Is that not the case?
You want to extremize $G(y):=\langle y,y''\rangle$ subject to the condition $\langle y,y\rangle = 1$, where $\langle u,v\rangle:=\int_0^1u(x)v(x)\,dx$ and $y(x)=a_1f_1(x)+a_2f_2(x)$. Using the method of Lagrange multipliers, we can recast this problem as the unconstrained extremization of \begin{align} \tilde{G}(y)&=G(y)-\lambda(\langle y,y\rangle - 1) =\sum_{i,j=1}^2(M_{ij}-\lambda\langle f_i,f_j\rangle)\, a_ia_j+\lambda. \tag{1} \end{align} To find the critical points of $\tilde{G}$, we have to solve the equations \begin{align} \frac{\partial\tilde{G}}{\partial a_1}&=0 \implies (M_{11}-\lambda\langle f_1,f_1\rangle)\, a_1 + (M_{12}-\lambda\langle f_1,f_2\rangle)\, a_2 = 0, \tag{2a} \\ \frac{\partial\tilde{G}}{\partial a_2}&=0 \implies (M_{21}-\lambda\langle f_2,f_1\rangle)\, a_1 + (M_{22}-\lambda\langle f_2,f_2\rangle)\, a_2 = 0. \tag{2b} \end{align} The system of equations $(2)$ has nontrivial solutions if $\lambda$ is a solution to the equation $$ \det(M-\lambda F)=0, \tag{3} $$ where $F$ is the matrix with elements $F_{ij}=\langle f_i,f_j\rangle$. If the functions $f_1, f_2$ are orthonormal, then $F$ is equal to the identity matrix, in which case the solutions to $(3)$ are the eigenvalues of $M$. In your problem, however, the functions $f_1, f_2$, although normalized to $1$, are not orthogonal; indeed, $\langle f_1,f_2\rangle=\sqrt{\frac{27}{28}}.$
You can verify that one of the solutions to $(3)$ coincides with the value that you found minimizing the Rayleigh quotient.