$u_t=8u_{xx}\\ u=u(t,x)\\ u(t,0)=u(t,\pi/2)=0\\ u(0,x)=h(x)\\ $ with $(t,x)$ in the domain $[0,\infty)\times[0,\pi/2]$
With this boundry value problem, I am trying to find the series solution.
My first question is how do you know if it has insulated ends or zero endpoint temperatures.
My second question is if the solution is constructed by the series with fourier sine/cos coefficients, how do you construct $a_n$ and $b_n$ when the nonhomogeneous condition is not further defined than $f(x)$?
Your system has zero temperature at its endpoints. The endpoints at located at $x = 0$ and $x = \frac{\pi}2$, and your boundary condition states that $u(t,0) = u(t, \frac{\pi}2) = 0$.
[If your system was insulated at its endpoints, the boundary condition would be $\partial_x u(t,0) = \partial_x u(t, \frac{\pi}2) = 0$, which says that there is no temperature gradient, and hence no heat flow, through the endpoints.]
When given a general $h(x)$ in the initial condition, you can expand $h(x)$ as a series, $$ h(x) = \sum_{n = 1}^\infty c_n \sin 2n x,$$ where $$ c_n = \frac{4}{\pi} \int_{0}^{\frac \pi 2} h(x) \sin 2n x dx$$ [In view of the $u(t,0) = u(t, \frac{\pi}2) = 0$ boundary conditions, I opted for a sine series.]
As you observed, separation of variables will produce a solution to the PDE involving constants $a_1, a_2, a_3, \dots$ that need to be determined. The best you can do is to relate these $a_n$'s to the $c_n$'s above.