I've solved the heat equation with Neumann boundary conditions on the rectangle $0 < x < a$ and $0 < y < b$, with initial condition $u(x,y,0)=f(x,y)$. The equation to solve was $$u_t = D^2 (u_{xx} + u_{yy}).$$
I performed separation of variables, with boundary conditions $X'(0)=X'(a) = 0$ and $Y'(0)=Y'(b) = 0$. The solution was $$u(x,y,t)=\sum_{m=1}^\infty \sum_{n=1}^\infty A_{m n}e^{-D^2 \pi^2 \big(\frac{m^2}{a^2}+\frac{n^2}{b^2} \big) t} \cos \left( \frac{m \pi}{a} x \right) \cos \left( \frac{n \pi}{b} y \right),$$
with $A_{mn} = \frac{4}{ab} \int^b_0 \int^a_0 f(x,y) \cos \left( \frac{m \pi}{a} x \right) \cos \left( \frac{n \pi}{b} y \right) dx dy $.
After this, we have
\begin{multline}u(x,y,t)= A_{00} + \sum_{m=1}^\infty A_{m 0} e^{- \frac{D^2 \pi^2 m^2}{a^2} t}\cos \left( \frac{m \pi}{a} x \right) + \sum_{n=1}^\infty A_{0 n} e^{- \frac{D^2 \pi^2 n^2}{b^2} t} \cos \left( \frac{n \pi}{b} y \right) \\ + \sum_{m=1}^\infty \sum_{n=1}^\infty A_{m n}e^{-D^2 \pi^2 \left(\frac{m^2}{a^2}+\frac{n^2}{b^2} \right) t} \cos \left( \frac{m \pi}{a} x \right) \cos \left( \frac{n \pi}{b} y \right)\end{multline}
Does the formula $A_{mn} = \frac{4}{ab} \int^b_0 \int^a_0 f(x,y) \cos \left( \frac{m \pi}{a} x \right) \cos \left( \frac{n \pi}{b} y \right) dx dy$ work for $A_{00}, A_{m0}$ and $A_{0n}$ ? Or do I need to repeat the process for every coefficient?
I believe the formula you're looking for is \begin{align} A_{mn} & = \frac{c_{mn}}{ab} \int_0^a \int_0^bf(x, y)\cos\left(\frac {m\pi}{a} x \right)\cos\left(\frac {n\pi}{n} y \right) \ dx dy, \end{align} where $$ c_{mn} = \begin{cases} \frac{1}{ab} & \text{if } m = 0 \text{ and } n = 0 \\ \frac{2}{ab} & \text{if } m \geq 1 \text{ and } n = 0 \\ \frac{2}{ab} & \text{if } m = 0 \text{ and } n \geq 1 \\ \frac{4}{ab} & \text{if } m \geq 1\text{ and } n \geq 1\\\end{cases}. $$
To derive these normalisation constants, I suggest you have a go at calculating the integral on the right-hand side of my formula for $A_{mn}$, using the series expansion \begin{align}f(x,y)= \sum_{m'=0}^\infty \sum_{n'=0}^\infty A_{m' n'} \cos \left( \frac{m' \pi}{a} x \right) \cos \left( \frac{n' \pi}{b} y \right).\end{align}
When calculating these integrals, you'll find it helpful to use the following identities. \begin{multline} \int_0^a \int_0^b \cos \left( \frac{m \pi}{a} x \right) \cos \left( \frac{n \pi}{b} y \right) \cos \left( \frac{m' \pi}{a} x \right) \cos \left( \frac{n' \pi}{b} y \right) \ dx dy \\ = \begin{cases} ab & \text{if } m = m' = 0 \text{ and } n = n' = 0 \\ \frac{ab}2 & \text{if } m = m' = 0 \text{ and } n = n' \geq 1 \\ \frac{ab}2 & \text{if } m = m' \geq 1 \text{ and } n = n' = 0 \\ \frac{ab}4 & \text{if } m = m' \geq 1 \text{ and } n = n' \geq 1 \\ 0 & \text{otherwise} \end{cases}. \end{multline}