My Question is: is there a binary operation on $(0, 1)$ that makes this set into an abelian group in which the inverse for any $x$ is $1-x$?
My approach to this was applying $f(x)=\pi(x-0.5)$ (which is bijective) so we map the given interval to $(-\pi/2, \pi/2)$.For elements $x, y, z \in (-\pi/2, \pi/2)$ we define
$$x \star y = z \Longleftrightarrow \tan(x) + \tan(y) = \tan(z).$$
This seems like a way to exploit the structure of $(\Bbb R, +)$. My problem is that I don't know how to define the group operation explicitly. I have tried to put, for $x, y \in (0,1)$, $$x \star y = f^{-1}(\arctan(\tan(f(x) + f(y))))$$ but this doesn't seem to be associative.
How do I find an explicit definition? Is this approach valid at all?
Consider the Sigmoid function $\sigma:\mathbb{R}\to (0,1)$ defined by $$\sigma(x) = \frac{1}{1+e^{-x}}=\frac{e^x}{1+e^{x}}$$ Note that $\sigma$ is invertible with $\sigma^{-1}(x)=\ln(\frac{x}{1-x})$. Moreover, $\sigma(0)=1/2$ and $$\sigma(-x) = \frac{e^{-x}}{1+e^{-x}} = 1-\frac{1}{1+e^{-x}}=1-\sigma(x)$$ Now define the binary operation $B$ on $(0,1)$ as follows. Given $x,y\in (0,1)$ $$ B(x,y) = \sigma\Big(\sigma^{-1}(x)+\sigma^{-1}(y)\Big) = \sigma\left(\ln \frac{xy}{(1-x)(1-y)}\right) = \frac{1}{1+\frac{(1-x)(1-y)}{xy}} $$ or in short $$ \boxed{B(x,y)=\frac{xy}{1-x-y+2xy}} $$ with inverse of $x$ becoming $1-x$ and identity being $1/2$.
More generally, let $f:\mathbb{R}\to \mathbb{R}$ be any odd strictly increasing continuous function with $\lim_{x\to \infty} f(x)=C<\infty$. Then construct $F(x)=(C+f(x))/2C$, which is $F: \mathbb{R}\to (0,1)$. Then you can define a binary operation on $(0,1)$ $$ B_f(x,y):= F\Big(F^{-1}(x)+F^{-1}(y)\Big) $$ making it an abelian group with inverse given by $x\mapsto 1-x$ and identity $1/2$. This means, you can equip this set with an abelian group structure (with desired properties) in infinitely (uncountably) different ways. In that sense, there is no "natural" choice, as there is no way of prefering one to another (unless you provide an additional structure).