Suppose $ u\in S^{'}(\mathbb{R}^{n}) $, where $ S^{'}(\mathbb{R}^{n}) $ is the dual space of the Schwartz-class functions $ S $; $ T^{*}u $ is the pullback via $ T $, i.e. if $ T: \mathbb{R}^{n}\to\mathbb{R}^{n} $ is a linear bijection and $ u\in S^{'}(\mathbb{R}^{n})\cap C(\mathbb{R}^{n}) $, we can define its pullback under $ T $ by $$ T^{*}u=u\circ T. $$
Note that a change of variables gives
$$ \begin{align} (T^{*}u)(\phi)&=\int u(Tx)\phi(x)dx\\ &= \int u(y)|\det T^{-1}|\phi (T^{-1}y)dy\\ &= u(|\det T^{-1}|\phi(T^{-1})), \end{align}$$ so, for general $ u\in S^{'}(\mathbb{R}^{n}) $, we define the pullback using the left and right sides of this equality.
With the above notation, prove that:
$$ \widehat{(T^{*}u)}=|\det T|^{-1}((T^{t})^{-1}))^{*}\hat{u}. $$
The Fourier transform is defined to be $$ \hat{f}(\xi)=\int_{\mathbb{R}^{n}}e^{-i<x, \xi>}f(x)dx $$ for $ f(x)\in L^{1}(\mathbb{R}^{n}) $.
Here is what I have: $$ \begin{align} (\widehat{T^{*}u})(\phi(x)) &= (T^{*}u)(\hat{\phi})\\ &=\int_{\mathbb{R}^{n}}T^{*}(u(x))\hat{\phi}(x)dx\\ &= \int_{\mathbb{R}^{n}}\widehat{T^{*}[u(x)]}\phi(x)dx\\ &= \int_{\mathbb{R}^{n}}\widehat{u[T(x)]}\phi(x)dx\\ &= \int_{\mathbb{R}^{n}}\left[\int_{\mathbb{R}^{n}}e^{-i<s, \xi>}u[T(s)]ds\right]\phi(x)dx\\ \end{align} $$
How to continue? I am wondering that where does the transpose come from?
You were off to a good start. For any $\phi \in \cal S(\mathbb R^n)$ the definitions provide $$\widehat{T^*u}(\phi) = \int_{\mathbb R^n} T^*u(x) \hat \phi(x) \, dx = \int_{\mathbb R^n} u(Tx) \hat \phi(x)\, dx.$$
Now change variables: with $y = Tx$ you get $dy = |\det T| dx$ so that $$ \int_{\mathbb R^n} u(Tx) \hat \phi(x)\, dx = |\det T|^{-1} \int_{\mathbb R^n} u(y) \hat \phi (T^{-1}y) \, dy.$$
Time to investigate the Fourier transform. Observe $$ \hat \phi (T^{-1} y) = \int_{\mathbb R^n} e^{-i \langle z,T^{-1}y \rangle} \phi(z) \, dz$$ and $\langle z,T^{-1}y \rangle = \langle (T^{-1})^tz,y \rangle = \langle (T^t)^{-1}z,y \rangle$, where the last equality uses the fact that the matrix representing $T$ is real.
Now combine the formulas. I will be a bit cavalier with the use of Fubini's theorem trusting that you can make it precise.
\begin{align*} \widehat{T^*u}(\phi) &= |\det T|^{-1} \int_{\mathbb R^n} \int_{\mathbb R^n} u(y) e^{-i \langle (T^t)^{-1}z,y \rangle} \phi(z) \, dz dy \\ &= |\det T|^{-1} \int_{\mathbb R^n} \int_{\mathbb R^n} e^{-i \langle (T^t)^{-1}z,y \rangle} u(y)\, dy \phi(z) \, dz. \end{align*}
As long as $u \in L^1(\mathbb R^n)$ (which doesn't seem to be implied by the hypotheses, but whatever) the inner integral equals $ \hat u ((T^t)^{-1}z) $ so that $$\widehat{T^*u}(\phi) = |\det T|^{-1} \int_{\mathbb R^n} \hat u ((T^t)^{-1}z) \phi(z) \, dz = |\det T|^{-1} \int_{\mathbb R^n} ((T^t)^{-1})^* \hat u(z) \phi(z) \, dz$$ where the last expression is $|\det T|^{-1} ((T^t)^{-1})^* \hat u (\phi).$ Finally disregard $\phi$ to obtain $$\widehat{T^*u} = |\det T|^{-1} ((T^t)^{-1})^* \hat u.$$