The full question is about a bit more than just commutativity, but I'm stuck on the commutativity part right now:
Let $R$ be a ring with the property that $f : R \rightarrow R, f(x) = x^2$, is a ring homomorphism of $R$ to itself. Prove: $R$ is commutative and $char(R) = 1$ or 2. Also prove that $1 + x \in R^*$ for all $x \in \ker(f)$
From the multiplicative property of homomorphisms it obviously follows that: $f(ab) = f(a)f(b) \implies abab = aabb$
And from the addition property we obtain: $f(a+b) = f(a) + f(b) \implies a^2 + ab + ba + b^2 = a^2 + b^2 \implies ab + ba = 0$
And also: $f(a+a) = f(a) + f(a) \implies a^2 + a^2 = 0$
I feel like I should be able to infer the commutativity just from what I've done here and I feel like I'm so close but I just can't seem to prove it no matter how I jumble things around.
If rings must have a $1$ (as suggested by the fact that you are also asked to prove that if $x\in \ker(f)$ then $1+x$ is a unit), then you have $$1+2a+a^2 = (1+a)^2 = f(1+a) = f(1)+f(a) = 1^2+a^2 = 1+a^2,$$ and therefore $2a=a+a=0$ for all $a$. This proves that the characteristic divides $2$, so the characteristic is either $1$ or $2$. In particular, $-x=x$ for all $x$. Since we already have that $ab+ba=0$, this gives $ba=-ab = ab$, so the ring is commutative.
If the ring doesn't have to have a $1$ then you cannot deduce characteristic $1$ or $2$: take $\mathbb{Z}$ with zero multiplication (usual addition, but $xy=0$ for all $x,y$); the characteristic is $0$, but $f$ is the zero map, which is both additive and multiplicative.
(Of course, that example is still a commutative ring (or rng); don't know off the top of my head if there is a noncommutative ring without unity in which $x\mapsto x^2$ is a ring homomorphism)