Let $(G, \cdot)$ be a group in which the functions $f: G \to G, f(x) = x^6$ and $g : G \to G, g(x) = x^{10}$ are endomorphisms and $f$ is injective. Prove that $G$ is an abelian group.
We need to prove that $f(xy) = f(yx), \forall x,y \in G$.
Because $f$ is an endomorphism, then $(xy)^6 = x^6y^6 \iff (yx)^5 = x^5y^5, \forall x,y \in G$. So, $x^6y^6 = (xy)^6 = (xy)^5(xy) = y^5x^5xy \implies x^6y^5 = y^5x^6, \forall x,y \in G$. Swaping $x$ and $y$, we get that $x^5y^6 = y^6x^5, \forall x,y \in G (*)$.
In the same way we obtain that $x^{10}y^9 = y^9x^{10}, \forall x,y \in G (**)$. Therefore, $x^5y^3 = y^3x^5$ (using $(*)$ and $(**)$), but I don't know how to continue from here.
Here is a solution assuming $f$ and $g$ are both injective, which is necessary as Derek points out.
Let us say a group is $(a,b)$-abelian for integers $a,b$ if $x^ay^b=y^bx^a$ for all $x,y\in G$. It is easily checked that if a group is $n$-abelian, it is $(n,n-1)$-abelian.
Next, we claim the following.
Lemma 1. Let $G$ be a group. If $G$ is $m$-abelian with $x\mapsto x^m$ an injective endomorphism, and $G$ is $(da,d'b)$-abelian where $d$ and $d'$ divide $m$, then $G$ is also $(a,b)$-abelian.
To see this, note that $G$ must be $(ma,mb)$-abelian, so that $$(x^ay^bx^{-a}y^{-b})^m= x^{ma}y^{mb}x^{-ma}y^{-mb}=1$$ for all $x,y\in G$ and thus by injectivity $G$ is $(a,b)$-abelian, this verifies the claim.
Now in our case, we have that $G$ is $(6,5)$-abelian, so applying the lemma for $m=6$, we have that $G$ is $(1,5)$-abelian. Applying the lemma again for $m=10$, we get that $G$ is $(1,1)$-abelian, hence the result.
Actually, there is no point in restricting to $6$ and $10$ so let us show the following.
To prove this, we need one more lemma.
Lemma 2. If $G$ is $(a,b)$-abelian and $(c,d)$-abelian, it is $(r,s)$-abelian where $r=\gcd(a,c)$ and $s=\mathrm{lcm}(b,d)$.
Indeed, choose integers $k,l$ such that $ka+lc=r$ and note that for all $x,y\in G$ we have $$x^{ak+cl}y^s=x^{ak}y^sx^{cl}=y^sx^{ak+cl}.$$
To prove the proposition, note that $G$ is $(m,m-1)$- and $(n,n-1)$-abelian, hence $(1,(m-1))$- and $(1,(n-1))$-abelian by lemma 1. Lemma $2$ implies $G$ is $(1,(m-1)(n-1))$ abelian and so applying lemma 1 several times we get the proposition.