Given a set defined as $S=\{(a,b) | a,b \in \mathbb{Q} \land a^2+b^2=1 \}$ and a binary operation $*$ defined as $(\forall(a,b),(c,d) \in S) ((a,b)*(c,d) = (ac-bd, bc+ad))$, determine whether $(S,*)$ is an Abelian group. I was able to prove that this structure is closed, associative , commutative and has neutral element. However, this structure doesn't seem to have a unique inverse element (or none at all) but I am not sure of this: Since neutral element is $e=(1,0)$, we can assume there is an inverse element such that: $$(a,b)*(x,y) = (x,y)*(a,b)= (1,0)$$ $$ax-by=1 \land bx+ay=0$$
How can I prove algebraically this system doesn't have a solution?
This is nothing more than the following system:
$$\left(\begin{array}{cc} a & -b \\ b & a\end{array}\right)\left(\begin{array}{c} x \\ y\end{array}\right) = \left(\begin{array}{c} 1 \\ 0\end{array}\right).$$
The inverse of a $2\times 2$ matrix $\left(\begin{array}{cc} a & b \\ c & d\end{array}\right)$ is nothing more than $\dfrac{1}{ad-bc}\left(\begin{array}{c} d & -b\\ -c & a\end{array}\right)$. Thus your system does in fact have a solution:
$$\left(\begin{array}{c} x \\ y\end{array}\right) = \frac{1}{a^2+b^2}\left(\begin{array}{c} a & b \\ -b & a\end{array}\right)\left(\begin{array}{c} 1 \\ 0\end{array}\right).$$
What you get then is that $x=a$ and $y=-b$ since $a^2+b^2=1$. This is not surprising since your group is really a subgroup of the complex numbers (with multiplication). In general, the inverse of a complex number $z$ is $\dfrac{z^*}{|z|^2}$. Since all of your elements live on the unit circle, $|z|=1$ so the inverse of $z$ is $z^*$.