Having the following boundary value problem: $$u_{xx}+u_{yy}=0,\quad 0<x<a,\; 0<y<b, \tag{1}$$ $$u_x(0,y)=u_x(a,y)=0,\quad 0\leq y\leq b,$$ $$u(x,0)= \cos{(\frac{\pi x}{a})},\;\; u(x,b)=\cos^2{(\frac{\pi x}{a})},\;\; 0\leq x\leq a.$$
I have done the following:
Using the method of separation of variables, the solution is of the form $u(x,y)=X(x)Y(y)$
$$(1) \Rightarrow X''Y+XY''=0 \Rightarrow \frac{X''}{X}+\frac{Y''}{Y}=0 \Rightarrow \frac{X''}{X}=- \frac{Y''}{Y}=- \lambda$$
So we get the following problems:
$$\left.\begin{matrix} X''+ \lambda X=0, 0<x<a\\ X'(0)=X'(a)=0 \end{matrix}\right\}\tag{*}$$
$(*) \Rightarrow $ The eigenvalues are
- $\lambda =0$ and the corresponding eigenvalue is $X_0(x)=1$;
- $\lambda_n =\left(\frac{n \pi}{a}\right)^2$ and the eigenfunctions are $ X_n(x)= \cos\left(\frac{n \pi x}{a}\right)$.
$$\left.\begin{matrix} Y_n''- \lambda_n Y_n=0,\quad 0<y<b\\ u(x,0)=\sum_{n=0}^{\infty} X_n(x)Y_n(0)= \cos{(\frac{\pi x}{a})}\\ u(x,b)=\sum_{n=0}^{\infty} X_n(x)Y_n(b)=\cos^2{(\frac{\pi x}{a})}\end{matrix}\right\} \tag{**}$$
$$(**) \Rightarrow Y_n(y)=\left\{\begin{matrix} a_0y+b_0, & n=0\\ a_n \sinh\left(\frac{n \pi y}{a}\right)+b_n \cosh\left(\frac{n \pi y}{a}\right) & n=1,2,3, \dots\,. \end{matrix}\right.$$
So $$u(x,y)=a_0y+b_0+ \sum_{n=1}^{\infty}{[a_n \sinh\left(\frac{n \pi y}{a}\right)+b_n \cosh\left(\frac{n \pi y}{a}\right) ] \cos{(\frac{n \pi x}{a})}}$$
By the condition $u(x,0)= \cos{(\frac{\pi x}{a})}$ we get: $b_0+ \sum_{n=1}^{\infty}{b_n \cos{(\frac{n \pi x}{a})}}=\cos{(\frac{\pi x}{a})}$
So $$b_0=0, b_1=1, b_n=0(n \neq 1)$$ $$\Rightarrow u(x,y)=a_0y+ \cosh\left(\frac{ \pi y}{a}\right) \cos{(\frac{n \pi x}{a})} +\sum_{n=1}^{\infty}{a_n \sinh\left(\frac{n \pi y}{a}\right) \cos{(\frac{n \pi x}{a})}}$$
By the condition $u(x,b)=\cos^2{(\frac{\pi x}{a})}$ we get: $a_0b+ \cosh\left(\frac{ \pi b}{a}\right) \cos{(\frac{n \pi x}{a})} +\sum_{n=1}^{\infty}{a_n \sinh\left(\frac{n \pi b}{a}\right) \cos{(\frac{n \pi x}{a})}}=\cos^2{(\frac{\pi x}{a})}$
So $$a_0=\frac{1}{2b}, a_1=-\coth{(\frac{\pi b}{a})}, a_2=\frac{1}{2 \sinh{(\frac{2 \pi b}{a})}},\; a_n=0\;(n \geq 3).$$
So the solution is: $$u(x,y)=\frac{1}{2b}y+[-\coth{(\frac{\pi b}{a})} \sinh{(\frac{\pi y}{a})}+\cosh{(\frac{\pi y}{a})}] \cos{(\frac{\pi x}{a})}+\frac{1}{2 \sinh{(\frac{2 \pi b}{a})}} \sinh{(\frac{2 \pi y}{a})} \cos{(\frac{2 \pi x}{a})}$$
Are the coefficients that I have found correct?? Is the solution of the problem correct??
I got the same final answer as you did. The only error I see is the typo $u(x,\color{blue}b)=\cdots=\cos^2(\pi x/a)$ which might have been addressed after my comment.