I am interested in finding the limit $t^2 \cos t x$ as $t \rightarrow \infty$ in the sense of schwartz distributions.
After some integration by parts I get
$$( t^2 \cos t x,\phi)=\int \cos(x)\phi(x/t)\frac{1}{t} \mathrm{d}x .$$
Can we from here conclude that the limit is zero by the dominated convergence theorem?
First note that you missed some derivatives on $\phi$ during your partial integration.
Furthermore, note $\frac{d^3}{dx^3}\left( -\frac{\sin(tx)}{t}\right) = t^2 \cdot \cos(tx)$ (i.e. use partial integration one more time). This leads to
$$\int t^2 \cdot \cos(tx) \cdot \phi(x) dx = \int \phi(x) \frac{d^3}{dx^3} \left(-\frac{\sin(tx)}{t}\right) dx = - (-1)^3 \int \phi'''(x) \cdot \frac{\sin(tx)}{t} dx.$$
Now note that the integrand is dominated by $\chi_{[-R,R]} \cdot \Vert \phi''' \Vert_{\sup} \in L^1$ for $t \geq 1$ (where $\rm{supp}(\phi) \subset [-R, R]$) and converges to zero pointwise.
Using dominated convergence, we conclude
$$\int t^2 \cdot \cos(tx) \cdot \phi(x) dx \xrightarrow[t \rightarrow \infty]{} 0.$$
In your integral, you can't really use dominated convergence, because for $t \rightarrow \infty$, the support of $\phi(x/t)$ will spread too much.