If $(f \circ g)(x)=h(x)$ and $(g \circ f)(x)=h^{-1}(x)$, is it necessary that $f(x)$ and $g(x)$ are inverses of each other? If not, what extra condition (other than just directly saying $h(x)=x$) must be imposed on $h(x)$ for them to be inverses? Assume that $f(x)$ and $g(x)$ are neither the identity function(s) nor equal to each other, and well-behaved (i.e. $C^{\infty}$ continuous and differentiable functions). All the 3 functions are invertible in their respective domains.
So, upon applying the inverse operator on the first equality,
$((f \circ g)(x))^{-1}=h^{-1}(x)$ $\implies$ $(g^{-1} \circ f^{-1})(x)=h^{-1}(x)$
$\implies (g \circ f)(x)=(g^{-1} \circ f^{-1})(x)$ and $(f \circ g)(x)=(f^{-1} \circ g^{-1})(x)$
However, I'm not able to proceed further and I haven't been able to come up with a counter-example yet. I was hoping to somehow demonstrate the commutativity of the composition of these two particular functions but that isn't easy to establish, just inspecting the above 2 relations. Does anyone have any ideas as to how to analyze these equations?
No, they don't have to be inverses of each other, and they don't have to commute.
An easy way to get counterexamples is to take two distinct involutions (functions equal to their own inverses). And there are lots of continuous involutions of real numbers.
For any real number $c$, let $f_c$ reflect the real line about $c$; that is, $f_c(x)=2c-x$. Note that $$f_c(f_c(x)) = f_c(2c-x) = 2c-2c+x = x.$$
Now pick $c\neq d$ and let $f=f_c$, $g=f_d$. Then $$(f\circ g)\circ(g\circ f)= f\circ(g\circ g)\circ f = f\circ f= \mathrm{id},$$ and likewise $(g\circ f)\circ(f\circ g) = \mathrm{id}$. So setting $f\circ g=h$ we have $g\circ f=h^{-1}$. But neither $f$ nor $g$ are the identity (they each have a unique fixed point), they are not inverses of each other, and they do not commute: $$\begin{align*} f\circ g(x)&=f(2d-x) = 2c - 2d + x,\\ g\circ f(x) &= g(2c-x) = 2d-2c +x. \end{align*}$$ Since $c\neq d$, then $2(c-d)\neq 2(d-c)$.