I have a problem, given $G$ is a non empty set and we know that $\star$ is associative binary operation on $G$ such that $a^2 \star b = b \star a^2$ for all $a,b \in G$. I need to proof that $G$ is abelian, I can proof $G$ is abelian but I have no idea how to proof G is a group first.
How do we proof $G$ is a group without knowing that binary operation $\star$ definition?
On
I don't think it is always a group. If there is a particular element $e$ in the set $G$, and for all $a$, $b$ in the set $a \star b=e$, then the binary operation is associative, and $a^2 \star b = b \star a^2 (= e)$ for all $a,b∈G$ but $(G, \star)$ is not a group (unless $G$ has only one element).
The assertion "G is a group" is not valid with these assumptions: take natural numbers $\mathbb{N}$ with the usual sum of integers. This operation is associative and $2a + b = b + 2a$ (addition is of course abelian, I've changed the given equation in your question to additive form) but ($\mathbb{N},+)$ is not a group.