Since the integral doesn't depend on $x$, I take $x^2$ out of the integral and take the derivative of a constant times a function. The final result is $x^2\sin(x^2)$.
I saw one suggested answer which gives a different answer: $x^2\sin(x^2)+2xf(x)$, which is the derivative using the product rule. Which one is correct? Please explain, thanks.
$x^2$ is a constant with respect to the integral (since it's an integral with respect to $t$) but $x^2$ isn't a constant when you differentiate with respect to $x$ (i.e. when you want to compute $f'(x)$).
Thus
$$f'(x) = \dfrac{d}{dx}\left(x^2\int_0^x\sin(t^2)dt\right) = \dfrac{d}{dx}\left(x^2\right)\underbrace{\int_0^x\sin(t^2)dt}_{=f(x)/x^2} + x^2\dfrac{d}{dx}\left(\int_0^x\sin(t^2)dt\right)$$
which gives $$f'(x) = 2x\dfrac{f(x)}{x^2} + x^2\sin(x^2) = 2\dfrac{f(x)}{x} + x^2\sin(x^2).$$