I solve these two problems below. In the first, $f(x)$ is monotonic. In the second, it isn't. I think the two solutions are the same procedure. But I'm told this solution method can only be used if $f(x)$ is monotonic. Why?
Problem $1$. Let $X$ be a random variable with PDF given by
$$f(x) = \begin{cases} 2x & \text{ if $0 < x < 1$}\\ 0 & \text{ otherwise.} \end{cases}$$
Let $Y = H(X) = e^{-X}$. Compute the PDF $g$ of $Y.$
Problem $2$. Let $X$ be a random variable with PDF given by $$f(x) = \begin{cases} \frac{1}{2} & \text{ if $-1 < x < 1$}\\ 0 & \text{ otherwise.} \end{cases}$$ Let $Y = H(X) = X^2$. Compute $g(y)$, the PDF $g$ of $Y$.
The first problem I can solve by $(1)$ computing the accumulated probability function $G(y)$ of the variable $Y$. Then $(2)$ I differentiate $G$ with respect to $y$, getting $g(y)$. Then $(3)$ I determine the domain of $g$, getting the entire definition of the PDF $g$ of $Y$.
I'm told this won't work for the second problem because $f(x)$ is not monotonic. Can you elaborate explaining what is the problem? Please use my example-solutions below if it helps.
Solution to problem $1$.
\begin{align*} G(y) &= P(Y \leq y)\\ &= P\left(\frac{1}{e^X} \leq y\right)\\ &= P\left(e^X \geq \frac{1}{y}\right)\\ &= P(X \geq \ln \frac{1}{y})\\ &= P(X \geq \ln 1 - \ln y)\\ &= P(X \geq -\ln y) &&\text{--- since $\ln 1 = 0$}\\ &= \int_{-\ln y}^\inf 2x\ dx &&\text{--- notice $0 < -\ln y$}\\ &= \int_{-\ln y}^1 2x\ dx &&\text{--- after 1, $f(x) = 0$}\\ &= x^2\rvert_{-\ln y}^1\\ &= 1 - (-\ln y)^2\\ &= 1 - (-\ln y)(-\ln y)\\ &= 1 - \ln^2 y, \end{align*}
as desired. Now differentiate $G(y),$ getting
\begin{align*} g(y) &= G'(y)\\ &= -2 \ln y \frac{1}{y}\\ &= -\frac{2}{y} \ln y, \end{align*}
as desired. The domain is $\frac{1}{e} < y < 1.$
Solution to problem $2$.
\begin{align*} G(y) &= P(Y \leq y)\\ &= P(X^2 \leq y)\\ &= P(|X| \leq \sqrt{y})\\ &= P(-\sqrt{y} \leq x \leq \sqrt{y}) &&\text{--- $F(x)$ from $-\sqrt{y}$ to $\sqrt{y}$}\\ &= P(x \leq \sqrt{y}) - P(x \leq -\sqrt{y})\\ &= F(\sqrt{y}) - F(\sqrt{y}). \end{align*}
Now differentiate, getting
\begin{align*} g(y) = G'(y)\\ &= f(\sqrt{y}) \frac{1}{2} y^{-\frac{1}{2}} - f(-\sqrt{y})\left(-\frac{1}{2}\right)y^{-\frac{1}{2}}\\ &= \frac{1}{4} \frac{1}{\sqrt{y}} + \frac{1}{4}\frac{1}{\sqrt y}\\ &= \frac{2}{4\sqrt y}\\ &= \frac{1}{2} y^{-\frac{1}{2}}. \end{align*}
Since $x \in (-1, 1)$ we get $y \in (-1,0) \cup (0,1).$
I don't see a difference in the solution-procedure. I feel I'm doing the same thing in each case.
There is no need for monotonicity of the PDF; not having that does not hurt anything. The issue is with monotonicity of the change of variable. The main point is that to compute the PDF of $G(X)$, you should pass through its CDF, meaning that you should compute $P(G(X) \leq x)$. If $G$ is strictly increasing, then this is $P(X \leq G^{-1}(x))$; if $G$ is strictly decreasing, then this is $P(X \geq G^{-1}(X))$*. If $G$ is neither strictly increasing nor decreasing, then this is $P(X \in G^{-1}((-\infty,x])$, and that preimage could take a wide variety of forms. For example, when $G(x)=x^2$, $G^{-1}((-\infty,x])=[-\sqrt{x},\sqrt{x}]$, which comes into play in the second problem.
* If $G$ has jumps, then it will not be surjective, but in this case there is a kind of "generalized inverse" that can be used. Let me know if you want clarification about this.