$S = \mathbb{Z}, a * b = a+b^2$
Commutative: $a*b = b*a$
$a*b = a + b^2$ and $b*a = b+a^2$ and they aren't the same at all.
Associative: $(a*b)*c = a*(b*c)$
$(a*b)*c = (a+b^2)* c = a+b^2+c^2$ and $a*(b*c) = a + (b+c^2)^2$ and they aren't the same at all. therefore it's not a binary operation on the set of integers. But the book says it is binary operation, I don't know where my mistake is in.
A binary operation $f$ on $\mathbb{S}$ is defined as an operation that takes $2$ values $a, b\in \mathbb{S}$ and returns a single value $c\in\mathbb{S}$, regardless of commutativity or associativity. For example, subtraction is a binary operation on $\mathbb{Z}$ because it always returns a value in $\mathbb{Z}$ even though it isn't commutative or associative. Applying these criteria, $*$ is a binary operation on $\mathbb{Z}$