Given two independent continuous random variables $X,Y$ on bounded intervals $[a,b]$ and $[c,d]$, respectively. Furthermore, let $h(x,y,z)$ be a non invertible function into $\mathbb{R}$, which is decreasing in its first argument and increasing in its second and third argument, i.e. $$\frac{\partial h}{\partial{x}}<0,\frac{\partial h}{\partial{y}}>0, \frac{\partial h}{\partial{z}}>0.$$ By non invertible I mean, that there is no way of writing down an inverse function for $h$.
No we define $\bar{z}(x,y)$ to be such that $h(x,y,\bar{z}(x,y))=0$ (no worries - this is well defined and exists).
Problem
Given the pdfs $f_X,f_Y$ of $X,Y$, I want to determine the pdf $f_\bar{z}$ of $\bar{z}$. I can plot the histogram/pdf with Matlab or any other tool, but I cannot tell anything analytically about the structure and curvature of the pdf. Regular solving techniques don't work due to $h$ being not "invertible". Any ideas of what I can say about $f_\bar{z}$?
A relation of $f_\bar{z}$ to $f_X$ or $f_Y$ would be terrific. An analytic derivation of $f_\bar{z}$ would be even more amazing.
Well, with just the details you gave there's not much to go on, but at least you can see that $\bar{z}(x,y)$ is increasing in $x$ and decreasing in $y$.
This suggests that there exist functions $\bar{x}(y,z)$ and $\bar{y}(x,z)$ such that $$ \bar{z}(\bar{x}(y,z),y) = z\\ \bar{z}(x,\bar{y}(x,z)) = z $$ or equivalently $$ h(\bar{x}(y,z), y, z) = 0\\ h(x, \bar{y}(x,z), z) = 0 $$ these answers will be unique if they exist, and where they don't you it's all right if we just let $\bar{x}(y,z)=a$ and $\bar{y}(x,z)=d$ respectively.
With these you can then prove that the following are all equivalent: $$ \def\d{\mathrm{d}} P(Z \leq t) = P(h(X,Y,t) \geq 0) = P(Y \geq \bar{y}(X,t)) = P(X \leq \bar{x}(Y,t)) $$
Which give the following expressions for the CDF of $Z$: $$ \begin{align} P(Z \leq t) &= \int_{\bar{z}(x,y) \leq t} f_X(x) f_Y(y) \, \d x \d y\\ P(h(X,Y,t) \geq 0) &= \int_{h(x,y,t) \geq 0} f_X(x) f_Y(y) \, \d x \d y\\ P(Y \geq \bar{y}(x,t)) &= \int_a^b f_X(x) \int_{\bar{y}(x,t)}^d f_Y(y) \, \d y \d x\\ P(X \leq \bar{x}(y,t)) &= \int_c^d f_Y(y) \int_a^{\bar{x}(y,t)}f_X(x) \, \d x \d y \end{align} $$
To get the PDF you can then differentiate these w.r.t. $t$.
If you really want to you can also write an expression for the PDF directly, but I wouldn't recommend it. For this you'll need to use that:
$$ \def\p{\partial} \frac{\p\bar{z}}{\p x} = - \frac{\p h / \p x}{\p h / \p z}, \quad \frac{\p\bar{z}}{\p y} = - \frac{\p h / \p y}{\p h / \p z}. $$
and then
$$ f_Z(z) = \int_{\bar{z}(x,y) = z} \frac{f_X(x) f_Y(y)}{\sqrt{\left(\frac{\p h / \p x}{\p h / \p z}\right)^2 + \left(\frac{\p h / \p y}{\p h / \p z}\right)^2}} \, \d x \d y $$
note that you'll still to find $x$ and $y$ that solve $\bar{z}(x,y) = z$, so I'm not sure if this will be much easier to solve than the expression for the CDF.