For any random variable that is a function of other random variables, e.g. $Z = g(X_1,X_2,X_3) = 5X_1 + (X_2X_3)^2$, is there a general formula/method to find $f_Z(z)$ given one knows $X_1,X_2,X_3$ are all independent and one also knows $f_{X_1}(x_1),f_{X_2}(x_2),f_{X_3}(x_3)$?
I know the existence of convolution i.e. to find $f_Z(z)$ when $Z=X_1+X_2+...+X_n$ but apart from convolution I know no other formula/method to find the probability function of a function of random variables.
On
This is a good question. I'd say the most general purpose technique is to find the CDF and then differentiate. I.e., calculate: $$F_Z(z) = \int_{(x_1,\dots,x_n):g(x_1,\dots,x_n) \leq z} f_{X_1,\dots,X_n}(x_1,\dots,x_n) \ dx_1\cdots \ dx_n$$ and then $$f_Z(z) = \frac{d}{dz}F_Z(z).$$ I think this will work in your particular example as long as you can determine the region of $(x_1,x_2,x_3)$ where $5x_1+(x_2x_3)^2 \leq z$. However, one can certainly come up with functions $g$ for which this won't be possible.
If $Z=g(X_i)$, then general formula would be: $$ f_Z(z) = \int \delta\left(g(x_i)-z\right)f_{X_i}(x_i)dx_i, $$ where $\delta(x)$ is Kronecker delta function and $f_{X_i}(x_i)$ is probability density function of vector $x_i$.
If $X_i$ has independent components, then joint pdf is a product of pdfs: $f_{X_i}(x_i)=f_{X_1}(x_1)f_{X_2}(x_2)\ldots f_{X_n}(x_n)$.
Finally, if it's possible to express one of the components $X_i$ from the function $g$, you can contract one of the integral and delta function: $$ f_{Z}(z) = \int\limits_{-\infty}^\infty\int\limits_{-\infty}^\infty\int\limits_{-\infty}^\infty \delta\left(5x_1+(x_2x_3)^2-z\right)f_{X_1}(x_1)f_{X_2}(x_2)f_{X_3}(x_3)dx_1dx_2dx_3=\\ \int\limits_{-\infty}^\infty\int\limits_{-\infty}^\infty \frac15f_{X_1}\left(\frac{z-(x_2x_3)^2}5\right)f_{X_2}(x_2)f_{X_3}(x_3)dx_2dx_3, $$