Use separation of variables to find a nonzero solution for the following Dirichlet problem:
$\Delta u = 0$ in $\Omega$ with $u=0$ on $\partial \Omega$, where $\Omega = \{(x,y) \in \mathbb{R}^{2}: 0 < y < \pi\}$
$\Omega \in \mathbb{R}^{2}$ is open and bounded, u is harmonic on $\Omega$.
If we assume that $u \in C^{2}(\Omega) \cap C(\bar{\Omega})$, then according to the maximum principle for harmonic functions we have:
$0 = \min_{y \in \partial \Omega} u(y) \leq u(x) \leq \max_{y \in \partial \Omega} u(y) = 0$ $\hspace{3mm} \forall x \in \Omega \hspace{3mm}$ (since $u = 0$ on $\partial \Omega$)
Hence u(x) = 0 for all $x \in \Omega$.
I have the feeling that there does not exist a nonzero solution, because of this maximum principle. So how do I know that there does exist a nonzero solution and how can I find this solution. I think I need to assume that there exist a solution $u(x,y) = X(x) \cdot Y(y)$. Then $0 = \Delta u = u_{xx} + u_{yy} = Y(y) \cdot X''(x) + X(x) \cdot Y''(y)$. So $X''(x) + aX(x) = 0$ and $Y''(y) + b \cdot Y(y) = 0$ for some constants $a = a(y)$ and $b = b(x)$. And then solve these ODEs for $X(x)$ and $Y(y)$. So $u(x,y) = X(x) \cdot Y(y) = (c_{1} \cdot \cos(\sqrt{a}x) + c_{2} \cdot \sin(\sqrt{a} x)) \cdot (d_{1} \cdot \cos(\sqrt{b}y) + d_{2} \cdot \sin(\sqrt{b} y))$
And we know that $u = 0$ on $\partial \Omega$. So that means that $u(x,0) = 0$ and $u(x,\pi) = 0$. And with these boundary values, I only find that $d_{1} = 0$.
So how to continue then? And what does this Dirichlet problem tell is then about the maximum principle and what is wrong about my conclusion that $u(x) = 0$ for all $x \in \Omega$?
The domain $\Omega$ is unbounded, in particular you do not know whether the solution u attains a maximum, which is an essential condition for the maximum principle to be applicable.
As you already observed, with the separation of variables Ansatz $u(x,y) = X(x)\cdot Y(y)$, the following formulas are obtained: $$ X(x) = c_1 \cos(\sqrt{a}x) + c_2 \sin( \sqrt{a}x), \quad Y(y)= d_2 \sin(\sqrt{b}y), $$ where we used $d_1 = 0$ (due to boundary condition). Now observe that $ a := \frac{Y''(y)}{Y(y)} = -b, $ assuming $b> 0$, we see that $$ X(x) = c_1 \cos(i\sqrt{b}x) + c_2 \sin(i\sqrt{b}x). $$ At this point, we have found a complex solution, in order to find a real solution, set $z = \sqrt{b}x$ and use the identity $ \cos(iz) -i \sin(iz) = e^z $ (i.e. we have to choose $c_1 = 1, c_2 = -i$). Thus we arrive at $ X(x) = e^{\sqrt{b}x}, $ and consequently $$ u(x,y) = e^{\sqrt{b}x} \cdot d_2 \sin(\sqrt{b}y). $$ Since $u_{xx} = bu, u_{yy} = -bu$, we see that u is a solution.