I have solved easily the (1) but I am really struggling with (2) and (3).
Do i need to use change of order of integration, because of so many parameters... i am confused a little bit.
My work:
$$\hat{Xf}(\xi,\theta)=\int\limits_{-\infty}^{\infty}\left(\int\limits_{-\infty}^{\infty}f(t\cos\theta+s\sin\theta,t\sin\theta - s\cos\theta) ds\right)e^{-2\pi i \xi t} dt = ??$$
How do I achieve $d\theta$, any hints please?
I got Jacobian as $t$ So, $$dxdy=t dt d\theta$$
I guess I am very close, I have also obtained $$\hat{f}(\xi\cos\theta,\xi\sin\theta)=\int\limits_{-\infty}^{\infty}\int\limits_{-\pi}^{\pi}f(t\cos\theta+s\sin\theta,t\sin\theta - s\cos\theta)e^{-2\pi i \xi t} t d\theta dt$$
For point (2): you correctly wrote the transform $\widehat{Xf}(\xi,\theta)$:
$$\widehat{Xf}(\xi,\theta)=\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}f(t\cos\theta+s\sin\theta,t\sin\theta - s\cos\theta)e^{-2\pi i \xi t} ds dt $$
Following the hint in the text, let $\gamma := (\cos(\theta),\sin(\theta))$ and $\gamma_\bot := (\sin(\theta),-\cos(\theta))$. It is easy to check that
$$ \xi t = \xi \gamma \cdot( t\gamma + s\gamma_\bot)$$ so replace this in the integral: $$\widehat{Xf}(\xi,\theta)=\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}f(t\gamma + s\gamma_\bot)e^{-2\pi i (\xi \gamma \cdot( t\gamma + s\gamma_\bot)) } ds dt $$
Now notice that $\gamma$ and $\gamma_\bot$ are a orthonormal basis for $\mathbb{R}^2$. Intuitively, this tells you that integrating over $ds dt$ is "the same" as integrating over $\mathbb{R}^2$ in standard coordinates. Formally notice that we can write
$$ \begin{bmatrix} x \\ y \end{bmatrix} = \underbrace{\begin{bmatrix} \cos(\theta) & -\sin(\theta) \\ \sin(\theta) & \cos(\theta) \end{bmatrix}\begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix}}_{A} \begin{bmatrix} t \\ s \end{bmatrix}$$
So using the formula for the multidimensional change of variables and knowing that $|\det(A)|=1$ we have
$$\widehat{Xf}(\xi,\theta)=\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}f(x,y)e^{-2\pi i \xi \langle \gamma, [x,y]\rangle } dx dy = \widehat{f}( \xi \gamma ) = \widehat{f}(\xi \cos(\theta), \xi \sin(\theta) ) $$
For point (3): knowing that $f\in \mathcal{S}$ we can write
$$ f(x,y) = \int_{\mathbb{R}^2} \hat f(a,b) e^{i 2\pi (ax+by)} da db $$
Writing the integral in polar coordinates and using (2):
$$ \begin{align*} f(x,y) & = \int_{-\pi}^{\pi} \int_{-\infty}^\infty \rho \hat f(\rho \cos(\theta), \rho \sin(\theta)) e^{i2\pi \rho (x\cos(\theta)+ y\sin(\theta))} d\rho d\theta = \\ & = \int_{-\pi}^{\pi} \int_{-\infty}^\infty \rho \widehat{Xf}(\rho,\theta) e^{i2\pi \rho (x\cos(\theta)+ y\sin(\theta))} d\rho d\theta =0 \end{align*}$$
where the last step follows from the fact that the Fourier transform of the constant function $0$ is identically null.