I have two functions with $a>0, b>1$:
\begin{align} f(x) &= ax \\ g(x) &= x - \frac{\tanh^{-1}(x)}{b}\end{align}
Now, the two functions always intersect at $(0, 0)$, and at two additional points for some specific values of $a,b$. I'm trying to find the conditions on $a, b$ such that there are three intersections. Here is a plot for such a case: f(x) in red, g(x) in green
I start with setting the two functions equal
$$ax = x - \frac{\tanh^{-1}(x)}{b}$$
but I'm totally unsure where to go from here. Various manipulations don't seem to lead to an easily interpretable expression.
\begin{align} ax = x - \frac{\tanh^{-1}(x)}{b} &\implies \tanh^{-1}(x)=b(1-a)x \qquad \text{, applies $\color{blue}{\tanh}$ to both sides}\\ &\implies x = \tanh(b(1-a)x) \end{align} Note that the argument of the R.H.S. $\color{blue}{\tanh(b(1-a)x)}$ is just $x$ scaled.
We know that $\color{blue}{\tanh(b(1-a)x)}$ is a sigmoid curve (S-shaped) with "rotational symmetry" $f(x) = -f(-x)$, which height increases yet saturates quickly $|\color{blue}{\tanh(b(1-a)x)} | < 1$, along with the slope decreasing when $|x|$ gets larger $\frac{d}{dx}\tanh(b(1-a)x) \to 0$. These follow directly from the definition and one can easily verify them in Desmos.
Therefore, if the slope at $x = 0$ is less than unity, which is the slope of $x$ on the L.H.S., then there's not going to be more intersection besides the origin $(0,0)$.
In other words, there will be exactly three intersections if and only if $$\frac{d}{dx}\tanh(b(1-a)x)\Big|_{x=0} > 1 \implies \frac{(1-a)b}{\cosh^2(b(1-a)x)}\Bigg|_{x=0} = (1-a)b> 1$$ Thus combined with the given condition, we have the three-intersection criterion:
$$b>\frac1{1-a} \quad \forall 0<a<1$$
One can easily verify this numerically in Desmos or other apps. For example, there's no way to intersect when $a > 1$. When $a = 0.8$ the (original two) curves start to intersect at $b = 5$.
BTW, the whole derivation could have been done in the "original form", before taking $\tanh$ on both sides (before turning $\tanh^{-1}$ into $\tanh$). The argument is literally the same, just that the derivative requires one more step.