For any $x \in (1,+\infty)$, $f(g(x))=x^2$, $g(f(x))=x^4$.
How to find the $f(x)$, $g(x)$ statisfying above functional equation.
Suppose inverse function existed.
The problem can be simplified to $f^2(x)=f(x^4)$.
How to find $f(x)$?
For any $x \in (1,+\infty)$, $f(g(x))=x^2$, $g(f(x))=x^4$.
How to find the $f(x)$, $g(x)$ statisfying above functional equation.
Suppose inverse function existed.
The problem can be simplified to $f^2(x)=f(x^4)$.
How to find $f(x)$?
On
Solving $$ f(x)^2 = f(x^4) . \tag{1}$$
We do not require that $f$ be continuous!
Let $f$ be arbitrary $[2,2^4) \to (1,+\infty)$. Extend recursively:
For $x \in [2^4,2^{4^2})$, let $f(x) = f(\sqrt[4]{x}\;)^2$. This means that $f$ satisfies $(1)$ for $x \in [2,2^{4})$.
For $x \in [2^{4^2}, 2^{4^3})$, let $f(x) = f(\sqrt[4]{x}\;)^2$. This means that $f$ satisfies $(1)$ for $x \in [2^4,2^{4^2})$.
Continue in this way: for $x>2^4$ define $f(x) = f(\sqrt[4]{x}\;)^2$.
In the other direction: For $x \in [2^{4^{-1}},2)$, let $f(x) = \sqrt{f(x^4)}$. This means that $f$ satisfies $(1)$ for $x \in [2^{4^{-1}},2)$.
For $x \in [2^{4^{-2}},2^{4^{-1}})$, let $f(x) = \sqrt{f(x^4)}$. This means that $f$ satisfies $(1)$ for $x \in [2^{4^{-2}},2^{4^{-1}})$.
Continue in this way: For $1 < x < 2$ define $f(x) = \sqrt{f(x^4)}$.
With $x=e^{4^t}$, $f^2(x)=f(x^4)$ turns to
$$2\log(f(e^{4^t}))=\log(f(e^{4^{t+1}}))$$ which is of the form
$$h(t+1)=2h(t),$$
and has the solution
$$h(t)=2^th_0.$$
So $$\log(f(x))=2^{\log_4(\log(x))}h_0=\sqrt{\log(x)}h_0$$
and finally,
$$f(x)=e^{\sqrt{\log(x)}h_0}=a_0^{\sqrt{\log(x)}}.$$
Technical note:
In fact, $a_0$ is not a constant. As the recurrence is given for values of $t$ one unit apart, $a_0(t)$ can be defined freely in any interval of unit length, such as $[t_0,t_0+1)$. This corresponds to a free definition of $f$ in $[x_0,x_0^4)$.