Apologies for the lengthy setup, but I want to make sure I am clear on how I am using the notation, and what I mean by the phrase "generalized number system".
Define a generalized number system $G$ as a set of elements equipped with
- addition operation, and $G$ is an abelian group under this operation (so addition is commutative and associative, and the existence and uniqueness of additive inverses also gives us subtraction)
- multiplication (and $G$ is closed under this operations)
- conjugate operation
$\overline{a}$ denotes the conjugate of $a$, with the properties that for all $a,b \in G$
$\overline{a} \in G$
$\overline{\overline{a}}=a$
$\overline{(a+b)}=\overline{a}+\overline{b}$
$\overline{(ab)}=\overline{b}\,\overline{a}$ - a linear order relation on the subset of elements for which $\overline{a}=a$
- multiplication and addition are "compatible", via the distributive property:
$a(b+c) = ab + ac$
Note that multiplication is not necessarily commutative, nor associative, nor provides a definition for division.
Side question: Is there a more mathematically appropriate term for what I am calling a generalized number system?
Now in the special case of $G$ being some field $F$, then given a vector space $V$ over $F$, we have an inner product space if we equip the vector space with a map
$\langle \cdot, \cdot \rangle : V \times V \to F$
which has the following properties for all $x,y,z \in V$ and $a \in F$:
$\langle x,y\rangle =\overline{\langle y,x\rangle}$
$\langle ax,y\rangle = a \langle x,y\rangle$
$\langle x+y,z\rangle = \langle x,z\rangle + \langle y,z\rangle$
$\langle x,x\rangle \geq 0$
$\langle x,x\rangle = 0 \Rightarrow x = 0$.
From this, an additional property can be derived:
$$\langle x,ay\rangle = \overline{\overline{\langle x,ay\rangle}} = \overline{\langle ay,x\rangle} = \overline{a \langle y,x\rangle} = \overline{\langle y,x\rangle}\,\overline{a} = \langle x,y\rangle \overline{a}$$
using that this is a field, we can further simplify:
$$\langle x,ay\rangle = \langle x,y\rangle \overline{a} = \overline{a} \langle x,y\rangle = \langle \overline{a}\,x,y\rangle$$
An "operator" on this vector space, will be defined as an map from the vector space back on itself
$\hat{A}: V \rightarrow V$
and the adjoint operator of $\hat{A}$ will be denoted $\hat{A}^\dagger$ is defined as the operator which satisfies:
$\langle \hat{A}x,y\rangle=\langle x,\hat{A}^\dagger y\rangle$
This is an important setup for considering self-adjoint operators in Hilbert space for quantum mechanics. Since people have played with quaternionic Hilbert spaces, there must be a way to generalize inner product spaces over non-commuting number systems (at least in some specific cases).
However, it is not clear to me how to generalize inner product spaces over non-commuting number systems. Even some of the defining properties require care, for now:
choosing the requirement on the inner product to be
$\langle ax,y\rangle = a \langle x,y\rangle$
would not be equivalent to choosing it to be
$\langle ax,y\rangle = \langle x,y\rangle a$
And furthermore, following the logic above, it does not appear to be possible to obtain:
$\langle x,ay\rangle = \langle \overline{a}\,x,y\rangle$
without commutative multiplication. Which appears to indicate even a very simple operator defined by linearly scaling a vector by a constant
(for all $x \in V$, $\hat{A}x = sx$ where $s \in G$), does not have an adjoint operator. So quite an amount of care must be taken when extending these concepts.
So how can one generalize inner product spaces to allow them to be over more general number spaces?
(This answer makes the additional assumption that multiplication is associative in your number system. There are difficulties even with this restriction. For more generality, you might want to look at composition algebras, for example.)
The number system with addition and multiplication as you describe it is called a ring. The conjugation you refer to is called an involution. (You should also demand that $\bar1=1$ if $1$ is the multiplicative identity.) Therefore your number system would generally be called an involutive ring. (In more generality, some kind of an involutive algebra.)
If you consider something like a vector space but replace the underlying field with a ring, the thing you end up with is called a module. In modules you have to pay attention to the direction from which you multiply my scalars. In an inner product it might be a good idea to consider it as a pairing between a left module and its opposite module, so that for a scalar $a$ you would have $\langle ax,y\rangle=a\langle x,y\rangle$ and $\langle x,ya\rangle=\langle x,y\rangle a$. If the module is commutative (multiplication is commutative but inverses need not exist), then things are somewhat simpler, but in general you will lose a lot of structure when going from vector spaces to modules.
There is one line in your question that I want to discuss: $\langle x,x\rangle\geq0$. This implicitly means that there is a meaningful order in the underlying ring or field. An ordered field necessarily has $\mathbb Q$ as a subfield, so all finite fields are excluded, for example. The complex field is not ordered, but the inner product is defined so that $\langle x,x\rangle$ is always real, so the condition makes sense. If you want this positivity condition for your inner product, make sure your underlying ring or field has an order.
The thing you seem to be after is something that I would call an inner product module, but I have not heard of such things being studied. It may well be that they do not have enough structure to make an interesting theory, but I don't know.