Could someone check if it's correct what I've done please?
Solve the following problem by separation of variables $$\nabla^2u(x,y)=0,\ \ 0\le x\le\pi, y\ge0\\ u(0,y)=u(\pi,y)=0\\ u(x,0)=\big(\frac{x}{\pi}\big )^2\\ \lim_{y\to\infty}u(x,y)=0$$
Note that I haven't applied the condition $\lim_{y\to\infty}u(x,y)=0,$ because it doesn't make sense, if $y\to\infty,$ then $u$ goes $\infty$ aswell a not to zero.
Note 2. This was an exam question however it's kinda complicated, calculating the coefficients $A_n$ for instance. Is there an alternative (easier) solution to this problem?
You've made a sign error. The general solution should be
$$ u(x,y) = \sum_n A_ne^{-\sqrt{\lambda_n}y}X_n(x) = \sum_{n=1}^\infty A_ne^{-n y}\sin(n x) $$
The indeed goes to zero as $y\to \infty$, since the $X$ part is always bounded. We set $B_n=0$ because of this boundary condition (the bounded solution is just an assumption that wasn't stated).
Also, you don't really need the constant $D_n$ there, since it would just combine with $A_n$ to make one constant anyway.
The last part is straightforward
$$ A_n = \frac{\int_0^\pi \left(\frac{x}{\pi}\right)^2\sin(n x)\ dx}{\int_0^\pi \sin^2(n x)\ dx} = \frac{2}{\pi}\int_0^\pi \left(\frac{x}{\pi}\right)^2\sin(n x)\ dx $$
This is (to me) the easiest part, since all you need to is solve the integral. Applying integration by parts twice gets it done. It's tedious, but not complicated.