I think a more appropriate tag would have been 'quasicomplex numbers' rather than 'hypercomplex numbers'.
I'm normally perfectly comfortable with the correspondence between hyperbolic functions & circular functions - the way one is just the other with imaginary argument, or an imaginary component in the argument ... & all that.
But I ran into a bit of a problem recently with this correspondence seeming to break down. I was looking at a post recently in which the matter of $$\sum_{k=1}^n\operatorname{atn}(x_k)$$$$=$$$$\operatorname{atn}\left(\frac{\sum _{k\inℕ_0,k<{n\over2}}(-1)^k\operatorname{het}_{n,2k+1}(\underline{x})}{\sum_{k\inℕ_0,k\leq{n\over2}}(-1)^k\operatorname{het}_{n,2k}(\underline{x})}\right)$$with $\operatorname{het}_{n,k}$ denoting the symmetric polynomial in $n$ variables of degree $k$ consisting of the sum of strictly heterogeneous products of the variables (I presume this is not standard notation; but I think it will do here). Another contibutor exposited that this is a consequence of the fact that $\operatorname{atn}x$ is the argument of $1+ix$ together with the fact that the argument of the product of a set of complex numbers is the sum of the arguments: and indeed if you expand$$\prod_{k=1}^n(1+ix_k)$$you get those symmetric polynomials, and the tangent of the argument is indeed the function of the $x_j$ constituting the content (it's getting awkward through 'argument' having two different meanings here!) of $\operatorname{atn}$ on the RHS of the relation shown above.
But you also have$$\sum_{k=1}^n\operatorname{atnh}(x_k)$$$$=$$$$\operatorname{atnh}\left(\frac{\sum _{k\inℕ_0,k<{n\over2}}\operatorname{het}_{n,2k+1}(\underline{x})}{\sum_{k\inℕ_0,k\leq{n\over2}}\operatorname{het}_{n,2k}(\underline{x})}\right) ,$$ and yet there is no corresponding logic for this ... unless ... perhaps you introduce an operator (let's call it $h$ for now) that has the properties $h≠1$, $|h|=1$, & $h^2=1$. The entity $h$ is not 1 ... and yet $h^2$ does (coz it does right!!?) = 1 ... $h$ is just eheieh asher eheieh - it just is what it is. (I've found that this point can be such a stumblingblock for people who are learning these things - the idea of an entity just being what it is ... & also how my own mind kept reverting to wanting some explicit statement in terms of $\pi$, or decimals, or whatever, of what $i$ is ... & how one day it just 'clicked' that it's not occasioned - and indeed there is none!)
And then you could have precisely analogous logic for the formula for the sum of plural $\operatorname{atanh}$, with the 'argument' of$$\prod_{k=1}^n(1+hx_k)$$ analogous to the argument of the circular complex product above.
The question is, is there any occurence of such ... hyperbolically complex numbers in mathematics atall; and has the idea ever been developed atall? I think if you were to put any kind of interpretation on the 'meaning' of this $h$, it would be that whereas $i$ is a rotation, $h$ is a reflection. And we are of course free to define any entity we wish ... the chief question then is "is the entity we have defined just a total cul-de-sac, or is there some mileage in it?" Maybe there is no mileage in this $h$ beyond it's being an expedient for buttressing the analogy between circular & hyperbolic functions.
Another entity we could define, for instance, and which in effect kind of is defined in the differential calculus would be (let's call it) $\epsilon$, which has the rather mutually-strange properties $\epsilon≠0$, $|\epsilon|=0$, & $\epsilon^2=0$.
As in the comments, these things exist and are called the split-complex numbers $\mathbb{R}[j]/(j^2 - 1)$. They are just another example of a ring, albeit a funny example because they aren't an integral domain and so some familiar properties of rings like the integers $\mathbb{Z}$, the rationals $\mathbb{Q}$, the real numbers $\mathbb{R}$, and the complex numbers $\mathbb{C}$ don't apply: for example, as you've observed, in the split complex numbers the equation $x^2 = 1$ has four solutions $1, -1, j, -j$, more than its degree.
In the same way as the complex numbers can be thought of in terms of $2 \times 2$ matrices via the correspondence
$$a + bi \leftrightarrow \left[ \begin{array}{cc} a & - b \\ b & a \end{array} \right]$$
the split-complex numbers can also be thought of in terms of $2 \times 2$ matrices via the correspondence
$$a + bj \leftrightarrow \left[ \begin{array}{cc} a & b \\ b & a \end{array} \right].$$
What this reveals is that, just as you wrote, in the same way that $i$ represents a rotation, $j$ represents a reflection.
Abstractly the split-complex numbers are isomorphic as a ring to the product $\mathbb{R} \times \mathbb{R}$ of two copies of the real numbers, via the correspondence
$$a + bj \leftrightarrow (a + b, a - b)$$
and so they aren't genuinely "new" in the same way that the complex numbers are. But again, as you say, they are a cute way of expressing the analogy between hyperbolic and trigonometric functions. For example there is a hyperbolic Euler's formula
$$\exp(j t) = \cosh t + j \sinh t.$$
The dual numbers $\mathbb{R}[\epsilon]/(\epsilon^2)$ as you mention, and as mentioned in the comments, are another example of a ring, and find some use. There are lots more rings than this, though. If you really want to have some fun you can learn about the quaternions next.