For random variable $x$, we have the PDF $f_x(x)$ and $y=g(x),x=h(y)$ which both are monotonic easily, there would be the PDF of y:
$$ f_y(y)=f_x(h(y))|h'(y)| $$
But what if $g(x)$ is not monotonic? For instance, $g(x)=J_1(x)$, in which $J_n(x)$ is the first kind of bessel function.
All I can come up with is numerical way or separate $g(x)$ into different monotonic parts. Is there any other equations or formulas made for non-monotonic $g(x)$?.
Any suggestion is appreciated! Thanks for reading!
You can try to seek for a partition $\langle A_i\rangle $ of the sample space $\mathcal X$ that makes the transformation monotonic. More formally, consider the functions $g_i$ defined on $A_i$ such that $g_i(x) =g(x) $ on $A_i, ~g_i$ is monotone in $A_i.$ Assuming they have the same range $\mathcal Y, $ if $g^{-1}$ has a continuous derivative in $\mathcal Y, $ then it is easy to see $$f_Y(y) =\begin{cases}\sum f_X\left(g_i^{-1}(y)\right)\left\vert\frac{\mathrm d}{\mathrm dy}g_i^{-1}(y))\right\vert&y\in\mathcal Y, \\0&\textrm{otherwise}\end{cases}$$
Even if the range isn't same, then one can proceed by taking $\mathcal Y_i$ as the range of $g_i.$ That is, for each $i, ~ g_i:A_i\to \mathcal Y_i$ is monotone.
Then, $$f_Y(y) =\sum \mathbf 1_{\mathcal Y_i}f_X\left(g_i^{-1}(y)\right)\left\vert\frac{\mathrm d}{\mathrm dy}g_i^{-1}(y))\right\vert.$$
This approach is in the line of Henry's advice.