I am reading an old paper C.Hopkins, "Non-abelian groups whose groups of isomorphisms are abelian", 1928.(Link: 1) I don't know what the term "operation" and "invariant" mean. The example phrases include:
A necessary and sufficient condition that an operation of the group of isomorphisms of an abelian group be invariant under this group is that it should transform every operation of this abelian group into the same power of itself.
Since the central quotient group of G is simply isomorphic with the group of inner isomorphisms of G, it is clear that every commutator of G is invariant under G.
The symbol G shall consistently denote a non-abelian group whose group of isomorphisms I is abelian; operations in G and operations of I shall be denoted by lower- case and capital letters respectively; an isomorphism of G with itself will often be called an automorphism.
Does the "operation" means a group action or an element of the group? Also, what does "invariant under the group G" mean?
"Operation" here means "element"; groups used to be thought of as collections of "operations on sets" (even after Cayley; this is the language used in Burnside's book, for example). So "operations of $G$" means "elements of $G$" and "operations of $I$" means "elements of $I$" (which here are automorphisms of $G$).
"Invariant subgroup" is an old term for "normal subgroup." So "invariant under this group" means that the element is fixed under conjugation; i.e., that it is central.
So what you quote would be expressed today perhaps as follows:
Do these make sense? I think so. It is easy to verify that automorphisms of an abelian group of the form $a\mapsto a^k$ (for fixed $k$) are central in $\mathrm{Aut}(A)$. The converse is, I believe, true but a bit trickier; I believe that for finite abelian groups it is the analog of the fact that the center of $\mathsf{GL}_n(F)$ consists of the scalar multiples of the identity...
For 2, since $G/Z(G)$ embeds into $\mathrm{Aut}(G)$, if $\mathrm{Aut}(G)$ is abelian, then $G/Z(G)$ is abelian, and therefore $[G,G]\subseteq Z(G)$.
3 is just notation.