My differentials prof taught us the convolution theorem and applied it to a differential equation
$ay'' + by' + cy = e^{t^{2}}$
Then he transformed it and left the transform as $L(e^{t^{2}}) = F(s)$
then changing it into $Y = \frac{F(s)}{G(s)}$ for some $G(s)$
following which he applied the convolution theorem to obtain the solution
What i don't understand is how can you work with the transform of some function using convolution theorem if the transform doesn't exist in the first place
Since $\mathcal{L}^{-1}\{\mathcal{L}\{f(t)\}\}=f(t)$ and $\mathcal{L}\{\mathcal{L}^{-1}\{F(s)\}\}=F(s)$ , in fact those relations can be extended even $\mathcal{L}\{f(t)\}$ or $\mathcal{L}^{-1}\{F(s)\}$ do not exist.
Moreover, I can also prove your professor did the particular result correctly.
According to the particular result your professor did, the full version of the general solution of the corresponding ODE done by variation of parameters is $y=C_1\sin t+C_2\cos t+\sin t\int_0^te^{t^2}\cos t~dt-\cos t\int_0^te^{t^2}\sin t~dt$
Note that the general solution can also rewrite as
$y=C_1\sin t+C_2\cos t+\sin t\int_0^te^{x^2}\cos x~dx-\cos t\int_0^te^{x^2}\sin x~dx$
$y=C_1\sin t+C_2\cos t+\int_0^te^{x^2}(\sin t\cos x-\cos t\sin x)~dx$
$y=C_1\sin t+C_2\cos t+\int_0^te^{x^2}\sin(t-x)~dx$
Which is consistent to the particular result your professor did.