I saw the problem from one math competition: $$\left(x^{3}-3x+1\right)\sqrt{x^{2}-1}+x^{4}-3x^{2}+x+1=0$$.
I tried to solve it this way: \begin{align*} & \left(x^{3}-3x+1\right)\sqrt{x^{2}-1}+x^{4}-3x^{2}+x+1=0\\ \Leftrightarrow\,\, & \left(x^{3}-3x+1\right)\sqrt{x^{2}-1}+x\left(x^{3}-3x+1\right)+1=0\\ \Leftrightarrow\,\, & \left(x^{3}-3x+1\right)\left(\sqrt{x^{2}-1}+x\right)=-1 \end{align*}
Now I tried to use substitutions like $a=x^3-3x$ and $b=\sqrt{x^{2}-1}$ but I cannot get anything. I know that solutions are $x=1$ and $x=\pm\sqrt{2}$.
A complete solution, building on your last identity.
Multiply your last equation by $x-\sqrt{x^2-1}\neq0$ to obtain:
$$\begin{align}x^3-3x+1&=\sqrt{x^2-1}-x\\x^3-2x+1&=\sqrt{x^2-1}\\x(x^2-2)+1&=\sqrt{(x^2-2)+1}\end{align}$$
$1$ is a solution by inspection of the second equation, and $\pm\sqrt{2}$ are solutions by inspection of the third equation, which was hopefully well motivated from a natural rearrangement.
Now taking $x$ not equal to one of the already determined solutions, square both sides, potentially introducing extraneous solutions: $$\begin{align}(x^3-2x+1)^2&=x^2-1\\x^6-4x^4+2x^3+3x^2-4x+2&=0\end{align}$$
We know that we can divide by $(x-1)$ and also by $x^2-2$, so we reduce to: $$(x-1)(x^5+x^4-3x^3-x^2+2x-2)=0\\(x^2-2)(x^3+x^2-x+1)=0$$
Now we know any solution of our original problem is either a solution already found, or it is a (real) root of $x^3+x^2-x+1=0$.
For such a root, the quantity $x^3-2x+1$ is equal to $-(x^2+x)$. Suppose $x\gt0$: then $0\gt0+1-x,x\gt1$ but this implies $0\gt1+1+1-x$ and $x\gt3$, so on, we have a contradiction in the magnitude of $x$. Moreover $x\gt3$ is impossible since Cauchy’s root bound ensures $|x|\le2$. Lets check the quantity $x^2+x$ - it is negative iff. $x\lt-x^2$, which forces that $-1\lt x$ by a comparison of magnitude. But then $0\gt-1+x^2-x+1=x^2-x$ and $x\gt x^2$, a contradiction. Thus the quantity $x^2+x$ is positive for all roots of the equation $x^3+x^2-x+1$. Were this root a solution of our desired equation, then $-(x^2+x)=\sqrt{x^2-1}$, but this is an impossibility as the left hand side is negative and the right hand side is positive. Thus this other root is extraneous and we can ignore it, and $1,\pm\sqrt{2}$ are the only solutions.