Let $X$ be a continuous RV with density function
$f_{X}(x)\begin{cases}
\frac{2}{9}x,& x\in [0,1] \\
0, &x \notin [0,1]
\end{cases}$
Let $Y=X(X-3)$. Determine the density of $Y$.
I am a bit with this problem especially given how $Y$ is defined and was wondering if someone could help me out. If you start by completing the square, i.e., $Y=(X-\frac{3}{2})^2-\frac{9}{4}$ would it take you anywhere? I have been trying with no result.
Attempt:
Let $g(X)=Y$. Then there is a theorem that states: If $X$ is a RV of the continuous type with PDF $f$, $y=g(x)$ is differentiable for all $x$ and strictly monotone, then $Y=g(X)$ is also a RV of the continuous type with PDF given by
$f_{Y}(y)\begin{cases}
f_{x}[g^{-1}(y)]|\frac{d}{dy}g^{-1}(y)|, &\alpha \lt y \lt \beta \\
0, & otherwise
\end{cases}$
where $\alpha=min\{g(-\infty),g(+\infty)\}$ and $\beta=\{g(-\infty),g(+\infty)\}$
$g(X)=Y=X(X-3)=(X^2-\frac{3}{2})^2-\frac{9}{4}$ is differentiable and invertible so it satisfies the conditions of the thm above. Therefore
$Y=g(X)$ is also a RV of the continuous type with PDF given by
$f_{Y}(y)\begin{cases}
f_{x}[(y+\frac{9}{4})^{\frac{1}{2}}+\frac{3}{2}]|\frac{1}{2}(y+\frac{9}{4})^{-\frac{1}{2}}|, & y \in [-2,0]\\
0, & otherwise
\end{cases}$
And then just fill out the details ...
$F_Y(y)=\mathbb P ( Y<y) = \mathbb P ( X(X-3)<y) = \mathbb P ( X^2-3X-y<0)$
Solve the polynomial equation $X^2-3X-y = 0$. If the roots are reals and distinct, then the polynomial expression is negative between the roots because the quadratic term is positive. Otherwise the probability is 0 and F_Y(y) = f_Y(y) = 0.
The roots are $x_+(y) = \frac{3+\sqrt{9+4y}}{2}$ and $x_-(y) = \frac{3-\sqrt{9+4y}}{2}$.
$9+4y>0 \iff y>-\frac 94$
Therefore
$\forall y>-\frac 94, \ \mathbb P ( X^2-3X-y<0) = \mathbb P ( x_-<X<x_+) = F_X(x_+(y))-F_X(x_-(y))$
$f_Y(y) = x_+'(y) f_X(x_+(y))-x_-'(y)f_X(x_-(y)) $
$\forall y<-\frac 94, \ f_y(y)=0$
After this point you just need to finish the computation. But there is an error in your density equation so I will save it for later.