Question
The solution manual said that $Y (y) = d_1 cosh (λ(b − y)) + d_2 sinh (λ(b − y))$. How did they get that solution?
My Work
$$Y''-λY = 0 \\ r = ± \sqrt\lambda \\ Y(y) = c_1e^{\sqrt\lambda y} + c_2e^{-\sqrt\lambda y} $$
As y $\rightarrow \infty$, $c_1e^{\sqrt\lambda y}$ becomes unbounded. Since $u(x,y)$ must be bounded, $c_1 = 0$.
From $X'' + \lambda X = 0$, I found that $\sqrt \lambda = \frac{n\pi}{a}$, so: $$Y_n = e^{-\frac{n\pi}{a} y} $$


Just to put the corrections to the cited solution in context: From more careful theory one gets that for $$u=\sum_n c_n X_n(x)Y_n(y)$$ to satisfy the boundary conditions, one needs already that the solutions $X_n,Y_n$ to $$Y_n''−λ_nY_n=0\text{ and accordingly }X_n''+λ_nX_n=0$$ need to satisfy the boundary conditions
At $y=0$ one gets the Fourier/sine series $$ h(x)=u(x,0)=\sum c_n\,\sinh\frac{n\pi b}a\,\sin\frac{n\pi x}a $$ which allows to compute the $c_n$ by computing sine resp. Fourier series coefficients of $h$ resp. its odd and then $2a$-periodic continuation.