Let $X_{ij} ~ N(\theta_i,\sigma^2)$ with $\sigma^2$ known, i = 1,... k, and j = 1, ... ,$n_i$. The prior distribution of $ \theta_i$ is $N(\phi,\tau^2)$, independently for i = 1,...,k and $(\phi,\tau^2)$ is has improper prior distribution given by $g(\phi,\tau^2)$= constant, with $ -\infty< \phi <\infty$ and $\tau>0$.
a) Let $\theta=(\theta_1,...,\theta_k)$. Find the joint posterior distribution of $\theta,\phi,\tau^2$ (up to a normalising constant
b) Deduce the posterior distribution of $\theta$.
I understand that for (b) we integrate the posterior $\pi(\theta,\phi,\tau^2|x)$ from (a) w.r.t. $\phi$ and $\tau^2$. However, the solution integrates w.r.t. $\tau$, not $\tau^2$. Why is this the case, and sceondly in general how do we know what function to integrate w.r.t. in such cases?
The reason why it matters whether we integrating over $\tau$ or $\tau^2$ lies in the theory of random variable transformation. In your case the original variable is $\tau$, even thought in the posterior it appears only in the form of $\tau^2$. If you want to treat $\tau^2$ as your variable you must adjust your posterior for this transformation.
Your posterior is $\pi\left (\theta, \tau, \phi \right )$. The solution given to you integrates over the parameter $\tau$ from $0$ to $+\infty$. Since in your posterior parameter $\tau$ only appears in the form of $\tau^2$ it is quite natural to consider $\tau^2$ as your new parameter. Therefore we perform a transformation of random variable, i.e. your new parameter is $\varphi\left ( \tau \right )=\tau^2$. Because you introduced a transformation your posterior distribution needs an adjustment:
$$\pi_n\left (\theta, \varphi, \phi \right )=\pi\left (\theta, \sqrt\varphi, \phi \right )\cdot\left | \frac{\mathrm{d} }{\mathrm{d} \varphi} \sqrt{\varphi}\right |$$
In other words just put $\varphi$ instead of $\tau^2$ in your posterior and multiply it by $\frac{1}{2\sqrt{\varphi}}$. Now you can integrate over your new variable $\varphi$.
I would strongly suggest reading some probability textbook containing general information about the random variable transformations to get a clear view.