The problem is to estimate the lowest eigenvalue of the following boundary value problem:
$$ u_{xx} + \lambda u = 0, \ \ \ \ x \in [0,1], \ \ \ \ u(0)=0, \ \ u_x(1)+u(1) = 0. $$
We are supposed to use the Rayleigh-Ritz method, with a trial function of the form $u=x^2 + \alpha x$.
https://i.stack.imgur.com/5KVC3.jpg
So far I have found the trial function to: $u(x)=x^2-1.5x$, by applying boundary conditions to the trial function.
My problem arises because when I use the Rayleigh Ritz method, $$F(u(x))=\frac{\int_0^1 p(x)u_x^2-q(x)u^2dx}{\int_0^1w(x)u^2dx}=\frac{\int_0^1u_x^2dx}{\int_0^1u^2dx} =\frac{\int_0^1(2x-1.5)^2dx}{\int_0^1(x^2-1.5x)^2dx}=2.92$$ I get an eigenvalue which is smaller than the exact value, $\lambda_{exact}=4.12$, which can't be right? Does it have something to do with a boundary term $[p(x)u_xu]_0^1$ somewhere not being equal to zero?
Yes, you're absolutely right. Since $[p(x)u_x u]_0^1 \neq 0$, you need to estimate the lowest eigenvalue using this formula: $$F[u(x)] = \frac{\int_0^1 \left[- u\frac{d}{dx} \left(p \frac{du}{dx}\right) - qu^2\right]dx}{\int_0^1 w u^2 dx} = \frac{-\int_0^1 u u_{xx} dx}{\int_0^1 u^2 dx}$$ [You can verify that this expression does indeed give an upper bound on the lowest eigenvalue by writing $u$ as a linear combination of (orthogonal) eigenfunctions - presumably this is done in your textbook.]
At first glance, you might believe that the expression I wrote down is equivalent to what you wrote down, by a familiar integration-by-parts trick. But this is not true! If you try to integrate the numerator by parts, you will pick up the non-zero term $-[p(x)u_x u]_0^1$.
Anyway, I quickly calculated this on Mathematica and got an answer of $4.17$, which is very close to the real answer!