Suppose we define the function G(x) = $\int_{0}^{x^2} [y^2 \int_{0}^{y} f(t)dt] dy$, and let's name $\phi(y) = y^2 \int_{0}^{y} f(t)dt$.
Then G'(x) = $\phi(x^2)*2x = 2x^5 \int_{0}^{x^2} f(t)dt$.
Going further, $\phi ' (x^2) = \frac{d}{dx} \left( x^4 \int_{0}^{x^2} f(t)dt \right) = 4x^3 \cdot \phi (x^2) + x^4\cdot f(x^2) \cdot 2x$.
Thus $G''(x) = \phi ' (x^2) \cdot 2x + 2 \phi (x^2) = 2x ( 4x^3 \cdot \phi (x^2) + 2x^5 \cdot f(x^2) ) + 2\phi (x^2).$
Is my math correct here? Did I apply the FTCII correctly?
You have applied the fundamental theorem correctly, however:
$$\begin{align}\phi'(x^2) &= \left.\dfrac{\mathrm d ~~}{\mathrm d y} \left(y^2\int_0^y f(t)\,\mathrm d t\right)\right\rvert_{y=x^2} \\[1ex]&= \left. 2y\int_0^y f(t)\,\mathrm d t+y^2 f(y)\right\rvert_{y=x^2}\\[1ex]&= 2x^2\int_0^{x^2} f(t)\,\mathrm d t+x^4 f(x^2)\\[1ex]&= 2x^{-2}{(x^2)}^2\int_0^{x^2} f(t)\,\mathrm d t+x^4 f(x^2)\\[1ex] &= 2 x^{-2}\phi(x^2)+x^4 f(x^2)\\[2ex] \dfrac{\mathrm d(\phi(x^2))}{\mathrm dx} &=2x\phi'(x^2)\\[1ex]&=4x^{-1}\phi(x^2)+2x^5f(x^2)\\[2ex] G''(x) &= \dfrac{\mathrm d(2x \phi(x^2))}{\mathrm d x}\\[1ex]&=2\phi(x^2)+2x\dfrac{\mathrm d(\phi(x^2))}{\mathrm d x}\\[1ex]&~~~\vdots\end{align}$$