I'm attempting to do problem (a) part (1).
Using separation of variables I managed to rewrite $u(x, t) = T(t)X(t)$ where $T(t) = Ae^{-(\lambda 2)^2t}$ and $X(x) = B\cos(\lambda x)$ and using the boundary conditions $u_x(0, t) = u_x(1, t) = 0$, I managed to show that $\lambda_n = n\pi$.
Using $\lambda_n$ this allows me to obtain functions $T_n(t) =Ae^{-(\lambda_n2)^2t}$ and $X_n(x) = B\cos(\lambda_n x)$
I'm not sure how to solve for the constants $A$ and $B$ however. How could I go about doing so?
Did you follow the full instructions? Parts $(3)$ and $(4)$ explicitly hint at what you should be doing.
Express the full solution as a series
$$ u(x,t) = \sum_{n=0}^\infty c_n e^{-(2n\pi)^2t} \cos(n\pi x) $$
Use the initial condition given
$$ u(x,0) = f(x) = \sum_{n=0}^\infty c_n \cos(n\pi x) $$
where $f(x) = x^2(1-x)^2$. This is a Fourier cosine series, so per orthogonality, we find
$$ c_0 = \int_0^1 f(x)\ dx $$ $$ c_n = 2\int_0^1 f(x)\cos(n\pi x)\ dx $$
Also note, there's only one set of constants (per $\lambda_n$) to be found here, since $A_n$ and $B_n$ combine in the multiplication.