If $S = \{x\in\mathbb{R} \mid x>0\}$ with the binary operation $\star$ given by
$$ x\star y = \sqrt{xy} $$
Then to show S is a group we of course need to show $(S,\star)$ satisfies the group axioms, the second of which is the associativity axiom:
(G$2$): For any $a,b,c\in S$, $a\star(b\star c) = (a\star b)\star c$
If we take $x,y,z\in S$ and apply this we get
$$x\star (y\star z) = \sqrt{x\sqrt{yz}} =\sqrt{x}\sqrt{\sqrt{y}} \sqrt{\sqrt{z}}$$
and $$(x\star y)\star z = \sqrt{z\sqrt{yx}}=\sqrt{z}\sqrt{\sqrt{y}} \sqrt{\sqrt{x}}$$
Am I being obtuse because I don't see how $x\star (y\star z)$ and $(x\star y)\star z$ could be equal here?