I need help with this question, I dont know what is wrong with this problem or is there any typo in the original question, By using $$u(x,y,t)=X(x)Y(y)T(t)$$ I am getting $$X(x)=0$$ which yields $$u(x,y,t)=0$$, But that should not be the case. All of trouble I am getting is because of the third boundary condition $u_x(a,y,t)=0$. What I feel it should be $u(a,y,t)=0$ instead of that $u_x$
Let me know how you will solve the question and obtain and non-zero answer ?
If you replace your guess $u(x, y, t) = X(x) Y(y) T(t)$ in the equation you'll get
$$ \frac{1}{X}\frac{{\rm d}^2X}{{\rm d}x^2} + \frac{1}{Y}\frac{{\rm d}^2Y}{{\rm d}x^2} = \frac{1}{\alpha^2 T}\frac{{\rm d}T}{{\rm d}t} = -k^2 $$
The sign of the constant just comes from the fact that a non-negative constant will make the temperature exponentially increase with time. So you have the equation
$$ \frac{1}{X}\frac{{\rm d}^2X}{{\rm d}x^2} = -k^2 - \frac{1}{Y}\frac{{\rm d}^2Y}{{\rm d}x^2} = -n^2 $$
or equivalentely
$$ \frac{1}{X}\frac{{\rm d}^2X}{{\rm d}x^2} = -n^2, ~~~~ \frac{1}{Y}\frac{{\rm d}^2Y}{{\rm d}Y^2} = -m^2, ~~~~ n^2 + m^2 = k^2 $$
The solutions of these are fairly straightforward, e.g.
$$ X(x) = A \cos n x + B \sin n x $$
with the conditions
\begin{eqnarray} X'(0) &=& 0 = -A n \sin (n\cdot 0) + B n \cos (n \cdot 0) = B n ~~~\Rightarrow~~~ B = 0 \\ X'(a) &=& 0 = -A n \sin (n\cdot a) = 0 \end{eqnarray}
This last equation has two possible solutions: $A = 0$, which makes $X(x) = 0$ and therefore $u(x, y, t) = 0$. And the other solution is
$$ n = \frac{n_x \pi}{a} ~~\mbox{for}~~~ n_x = 0, 1,\cdots $$
You can repeat the analysis for $Y(y)$