I have to find $f_Y$ if $f_X$ is given as $f_X=30x^2(1-x)^2, 0 <x < 1$ and $Y=X^2$. How do I do this?
Ok, then I get: $F_Y=P(x \le y^{\frac{1}{2}}) \Rightarrow F_X(y^{\frac{1}{2}})$
$F(y)=\int^{\sqrt{y}}_0 30x^2(1-x)^2dx=\left[ u'=30x^2 \Rightarrow u=10x^3 \\ v=(1-x)^2 \Rightarrow v'=-2(1-x) \right]=\left. 10x^3 (1-x^2) + 60\int x^3(1-x^2)dx=10x^3-10x^2+15x^4-\frac{60}{7}x^7 \right|^{\sqrt{y}}_0=10y^{\frac{3}{2}}-10y+15y^2-\frac{60}{7}y^3\sqrt{y}$
IS at least $F(y) $ correctly calculated?
$$F_Y(y)=P(X^2\le y)=P(-\sqrt y\le X\le\sqrt y)=P(X\le\sqrt y)=F_X(\sqrt y)$$
Since
$$f_X(x)=\begin{cases}30x^2(1-x)^2&\text{for }0<x<1\\0&\text{otherwise}\end{cases}$$
the CDF for $X$ is
$$F_X(x)=\int_{-\infty}^xf_X(t)\,\mathrm dt=\begin{cases}0&\text{for }x<0\\10 x^3 - 15 x^4 + 6 x^5&\text{for }0\le x<1\\1&\text{for }x\ge1\end{cases}$$
so that the CDF for $Y$ is
$$F_Y(y)=F_X(\sqrt y)=\begin{cases}0&\text{for }y<0\\10y^{3/2} - 15y^2 + 6 y^{5/2}&\text{for }0\le y<1\\1&\text{for }y\ge1\end{cases}$$
and hence its PDF is
$$f_Y(y)=\frac{\mathrm dF_Y}{\mathrm dy}=\begin{cases}15\sqrt y-30y+15y^{3/2}&\text{for }0<y<1\\0&\text{otherwise}\end{cases}$$
You made a mistake while integrating $f_X(x)$. Integrating by parts gives
$$\int_0^{\sqrt y}30x^2(1-x)^2\,\mathrm dx=vu\bigg|_0^{\sqrt y}-\int_0^{\sqrt y}u\,\mathrm dv$$
where
$$\begin{matrix}v=(1-x)^2&\mathrm dv=-2(1-x)\,\mathrm dx\\\mathrm du=30x^2\,\mathrm dx&u=10x^3\end{matrix}$$
$$\begin{align*} \int_0^{\sqrt y}30x^2(1-x)^2\,\mathrm dx&=10x^3(1-x)^2\bigg|_0^{\sqrt y}+\int_0^{\sqrt y}20x^3(1-x)\,\mathrm dx\\[1ex] &=10y^{3/2}(1-\sqrt y)^2+(5x^4-4x^5)\bigg|_0^{\sqrt y}\\[1ex] &=10y^{3/2}(1-2\sqrt y+y)+5y^2-4y^{5/2}\\[1ex] &=10y^{3/2}-15y^2+6y^{5/2} \end{align*}$$