Take some function such as $f(x)=x\sin x+3x-1$, or $g(x)=xe^x-2x+3$. These functions don't have a closed-form inverse (I know this because in general we are told that these should be solved numerically, as there is no analytic way of solving equations such as $f(x)=0$).
Why do they not have a closed-form inverse? Is there any way of justifying this?
On
Yes, there is a way of justifying this.
"Closed form" means expressions of allowed functions (Wikipedia: Closed-form expression). If an equation is solvable in closed form depends therefore on the functions you allow.
For the elementary functions, there is a structure theorem that can help to decide if a given kind of equations of elementary functions can have solutions that can be expressed as function value of an elementary function.
The elementary functions are according to Liouville and Ritt those functions of one variable which are obtained in a finite number of steps by performing algebraic operations and taking exponentials and logarithms (Wikipedia: Elementary function).
The incomprehensibly unfortunately hardly noticed theorem of Joseph Fels Ritt in Ritt, J. F.: Elementary functions and their inverses. Trans. Amer. Math. Soc. 27 (1925) (1) 68-90 answers which kinds of Elementary functions can have an inverse which is an Elementary function. You can also take the method of Rosenlicht, M.: On the explicit solvability of certain transcendental equations. Publications mathématiques de l'IHÉS 36 (1969) 15-22.
For the functions $f$ you gave, no expln-form of Ritt exists that has the structure $A(f_{1}(x))$, where $f_{1}$ is a transcendental function and $A$ is an algebraic function of only one variable. Therefore both $f$ cannot have an elementary inverse, according to the theorem of Ritt.
Another method is decribed in Khovanskii, A.: Topological Galois Theory. Solvability and Unsolvability of Equations in Finite Terms. Springer 2014 and articles of A. Khovanskii and Y. Burda. It is applied in the following article.
Belov-Kanel, A.; Malistov, A.; Zaytsev, R.: Solvability of equations in elementary functions. Journal of Knot Theory and Its Ramifications 29 (2020) (2) 204-205
see also: How can we show that $A(z,e^z)$ and $A(\ln (z),z)$ have no elementary inverse?
I suppose all of these methods could be extended to other classes of closed-form functions.
If $f$ can be decomposed into compositions of algebraic functions and other known Standard functions than $\exp$ and $\ln$, an analog theorem to the theorem of Ritt of [Ritt 1925] could be applied. I hope to prove such a generalization of Ritt's theorem for this class of functions.
Briefly, because the equations you'd need to solve to get a closed form of this inverse are transcendental equations. You might also find the following article helpful.
Edit: Some more info - it might seem that if you end up with an equation of the form $x=h(x)$ where $h$ is a transcendental function, then you're out of luck and can't find a closed-form solution for $x$. However, this is not the case, for example $$\sin x = x$$ can be completely solved by eyeballing $x=0$ as the only solution (and this is admittedly a closed-form way to settle things!). The point is that transcendence is more correctly understood as preventing the existence of closed-form solutions for a class of equations instead of for a single equation. This is similar to the sense in which there is no general formula involving only the elementary algebraic operations and taking radicals to solve a polynomial equation of degree $5$ or more, and yet we can deal with many special cases (and indeed entire sub-classes of such polynomial equations).
So, briefly: transcendence is no magical excuse to stop thinking :) and with practice and over time you will develop a feeling for which equations are hopeless and which not so much.