My question differs from this one(also, the binary operation differs slightly). Here, I'm supposed to determine $G$, as opposed to proving a given $G$ to be an Abelian group. I know that for $(G,*)$ to be a group, it has to satisfy the following postulates:
1) Closure, i.e., $a*b\in G$, $\forall a,b\in G$
2) Associativity, i.e., $(a*b)*c=a*(b*c)$, $\forall a,b,c\in G$
3) Existence of a unique identity element $e\in G$ such that $a*e=e*a=a$, $\forall a\in G$
4) Existence of inverse elements, i.e., $\forall a\in G$, $\exists a^{-1}\in G$ such that $a*a^{-1}=a^{-1}*a=e$
I begin by assuming $G$ to be identical to $\Bbb R$, because I have no idea how to proceed with an arbitrary subset of $\Bbb R$ whose elements and properties are unknown. Since addition and multiplication are both closed on $\Bbb R$, the closure property is satisfied by $(G,*)$. $(G,*)$ is also associative(as can be shown by trivial calculations). Calculations prove that $0$ is the identity element. So far so good. Now to evaluate the existence of inverse elements, I use postulate 4 $$a*a^{-1}=e$$ This means $$a+a^{-1}+aa^{-1}=0$$ This equation proves that $a^{-1}$ does not exist when $a=-1$. Intuitively, it seems reasonable that $G=\Bbb R\backslash \{-1\}$. However, this requires to check the consistency of the first 3 postulates again. I'm not sure if I can rely on addition and multiplication being closed operations on $\Bbb R$ anymore, now that one of the elements has been excluded. I'm confused as to how to proceed forward and seek guidance for the same.
Edit: Apparently I overlooked the postulate of commutativity for Abelian Groups, but as it stands I think proving $(G,*)$ to be a group is the primary challenge for me.
You first need to determine what is $e$: $e*e=e$, so $e+e+e^2=e$, so $e^2=-e$. Hence $e=0$ or $e=-1$. Then $e*x=x$, so $e+x+ex=x$ or $e+ex=0$ for every $x\in G$. If $e=-1$, then the only $x$ is $x=-1$ and so $G=\{-1\}$. So we can assume $e=0$. Then for every $x\in G$ there should be $y$: $x*y=0$, that is $x+y+xy=0$.So $y=1/(x+1)-1=x/(x+1)$. So with every $x\in G$ this $y$ should be in $G$ too. Finally $G$ must be associative which gives some more restrictions on $G$: $$(a*b)*c= (a+b+ab)*c=a+b+ab+c+ac+bc+abc=a*(b*c)=a*(b+c+bc)=a+b+c+bc+ab+ac+abc,$$ so for every $a,b,c\in G$: $0=0$ - no new restrictions. Hence the only restrictions on $G$ are
Either $G=\{-1\}$ or $0\in G, -1\not\in G$;
For every $a,b\in G$, $ab+a+b\in G$;
If $0\in G$ then for every $x\in G$, $x/(x+1)\in G$.
For example $G$ can be one of the following sets
a) $\mathbb{Q}\setminus \{-1\}$;
b) $\mathbb{R}\setminus\{-1\}$;
c) $\mathbb{Q}_{\ge 0}$
but also many other subsets of $\mathbb{R}$ (continuously many).