For real $x>1$, the function $f(x)$ is increasing and maps the interval $(1,+\infty)$ onto $(-\infty,+\infty)$, so that we may define inverse function $f^{\text{-1}}:\mathbb R \to (1,+\infty)$ by
$$ f^{\text{-1}} (f(x))=x$$
is it possible to determine $$ f^{\text{-1}} (f(x)^2) ?$$
There is no way to simplify the expression $f^{-1}(f(x)^2)$ without more specific information about $f$.
For example, maybe $f(x)=\ln(x-1)$, which meets your conditions. Then $f^{-1}(f(x)^2)=e^{\ln(x-1)^2}+1=(x-1)^{\ln(x-1)}+1$. Is there some way to write that in terms of "$f(x)$" and $f^{-1}(x)$" that is simpler than $f^{-1}(f(x)^2)$?
For another example, maybe $f(x)=\frac{x^2-2x}{x-1}$, which meets your conditions. Then $f^{-1}(f(x)^2)=1+\frac12\left(\frac{x^2-2x}{x-1}\right)^2+\frac12\sqrt{4+\left(\frac{x^2-2x}{x-1}\right)^4}$. Is there some way to write that in terms of "$f(x)$" and $f^{-1}(x)$" that is simpler than $f^{-1}(f(x)^2)$? Consistent with the previous example?