Let $\lambda_1, \lambda_2$ be the eigenvalues and $f_1 , f_2$ be the corresponding eigenfunctions for the homogeneous integral equation:
$$ \phi(x) - \lambda \int_0^1 (xt +2x^2) \phi(t) dt = 0 $$
Then:
$\lambda_1 = -18 - 6 \sqrt{10} , \lambda_2 = -18 + 6 \sqrt{10}$
$\lambda_1 = -36 - 12 \sqrt{10} , \lambda_2 = -36 + 12 \sqrt{10}$
$\int_0^1f_1(x)f_2(x) dx = 1$
$\int_0^1f_1(x)f_2(x) dx = 0$
I solved simply and got that the values of $\lambda_1$ and $\lambda_2$ belong to (1).
Please tell me about options (3) and (4).
Thank you.
This work is almost completed.
Define $g(t)=\int_0^t \phi(s) ds$ and $h(t)=\int_0^t s \phi(s) ds$ Because $\phi(0)=0$, we have $g(0)=0$ and $h(0)=0$.
Using integration by parts, we have
$$h(t)=\int_0^t t \phi(t) dt=t\left(\int_0^t\phi(s) ds\right)-\int_0^t\left(\int_0^s \phi(r) dr\right)ds$$ $$\implies h(t)=tg(t)-\int_0^t g(s) ds......(1)$$
The original equation becomes: $$\phi(x) = \lambda x\ h(1) +2 \lambda x^2 g(1)......(2)$$
And $$g(t)=\int_0^t \phi(s) ds=\frac{1}{2}\lambda h(1)t^2 +\frac{2}{3} \lambda g(1) t^3......(3)$$
$$h(t)=\int_0^t s\phi(s) ds=\frac{1}{3}\lambda h(1)t^3 +\frac{2}{4} \lambda g(1) t^4......(4)$$
Therefore
$$g(1)=\frac{1}{2} \lambda h(1)+\frac{2}{3} \lambda g(1)...(5)$$
$$h(1)=\frac{1}{3}\lambda h(1) +\frac{2}{4} \lambda g(1)......(6)$$
Setting $\mu=\lambda^{-1}$ in (5) and (6), we have: $$ \left(\begin{matrix} \frac{2}{3} & \frac{2}{3} \\ \frac{2}{4} & \frac{1}{3} \\ \end{matrix}\right) \left(\begin{matrix} g(1) \\ h(1) \\ \end{matrix}\right)= \mu \left(\begin{matrix} g(1) \\ h(1) \\ \end{matrix}\right)......(7) $$
Once you found two eigen values you can substitute them into (7) and solve for $g(1)$ as a function of $h(1)$. You may then fix $h(1)$ by some kind of normalization.
Finally you can substitute $\lambda_{1,2}$ and $g(1),h(1)$ into (2) to obtain the eigen functions $f_1(x)$ and $f_2(x)$.
I hope that you can pick it up from here.
-mike