Consider the problem $$ \begin{eqnarray} \Delta u=0 \\ \frac{\partial u}{\partial r}(a,\theta) = 0 \\ u(b,\theta)=f(\theta) \end{eqnarray}\tag1$$
Using method of separable variable solve the problem.
My attempt:
Let $u(r,\theta)=h(r)P(\theta)\tag2$ Then substituting $(2)$ in $(1)$ and dividin we have $$h''(r)P(\theta)+\frac{1}{r}h'(r)P(\theta)+\frac{1}{r^2}h(r)P''(\theta)=0$$
Dividing for $h(r)P(\theta)$ and we have: $$r^2\frac{h''(r)}{h(r)}+r\frac{h'(r)}{h(r)}+\frac{P''(\theta)}{P(\theta)}=0$$
This implies:
$$\frac{d}{d\theta}(\frac{P''(\theta)}{P(\theta)})=0\implies \frac{P''(\theta)}{P(\theta)}=-\lambda$$
Then: $$r^2\frac{h''(r)}{h(r)}+r\frac{h'(r)}{h(r)}=-\frac{P''(\theta)}{P(\theta)}=\lambda$$
Then we have two equations: $$ \begin{eqnarray} P''(\theta)+\lambda P(\theta)=0 \\ r^2h''(r)+r{h'(r)}-\lambda h(r)=0 \\ \end{eqnarray}\tag3$$
Here i'm stuck. Can someone help me?
In order to satisfy the periodicity condition for $P(\theta)$, you need $\lambda = n^2$ where $n = 0,1,2,.\dots$ (check this), so the equation for $h(r)$ becomes
$$ r^2 h'' + rh' - n^2 h= 0 $$
which is a Cauchy-Euler equation. The solution has the form
$$ h_n(r) = \begin{cases} A_0 + B_0\ln r, & n = 0 \\ A_nr^n + B_n r^{-n}, & n > 0 \end{cases} $$
If you apply the boundary condition $r'(a) = 0$ for each case, you'll arrive at (up to a multiplicative constant)
$$ h_n(r) = \begin{cases} 1, & n = 0 \\ \left(\dfrac{r}{a}\right)^n + \left(\dfrac{r}{a}\right)^{-n}, & n > 0 \end{cases} $$
By linearity, the general solution is
$$ u(r,\theta) = C_0 + \sum_{n=1}^\infty \left[\left(\frac{r}{a}\right)^n + \left(\frac{r}{a}\right)^{-n}\right]\big[C_n\cos(n\theta) + D_n\sin(n\theta)\big] $$
The remaining boundary condition gives
$$ u(b,\theta) = f(\theta) = C_0 + \sum_{n=1}^\infty \left[\left(\frac{b}{a}\right)^n + \left(\frac{b}{a}\right)^{-n}\right]\big[C_n\cos(n\theta) + D_n\sin(n\theta)\big] $$
Compare this with the Fourier series of $f(\theta)$ in $(0,2\pi)$ to determine the constants $C_n$, $D_n$