Throughout this question I will presume the following: $G$ is a finite group generated by two elements $a,b$, and $f: \{a,b\} \rightarrow G$ is a function, such that $f(a),f(b)$ generate $G$.
-
Because $a,b$ generate $G$, it's clear that if $f$ can be extended to an automorphism, it can only be done so in a unique way.
a - Is there some neat way of seeing whether $f$ induces an automorphism?
b - If $a$ and $b$ are of the same order, when does $a \mapsto b$ and $b \mapsto a$ induce an automorphism? What if $\langle a \rangle \cap \langle b \rangle = \emptyset$?
-
Some of my notes:
It's enough to check that $f$ induces a homomorphism.
For commutative groups the question is easy - if $f$ preserves orders, then it should induce an automorphism.
A note on the second question b - generally, preserving orders isn't enough - I'm not quite I sure I got this right, but looking at the group with presentation $\langle\,a,b\mid a^4=1\,, b^4=1 \,, ab=ba^3 \rangle$, definifing $f$ by $a \mapsto b$ and $b \mapsto a$ should not induce an automorphism - we use the fact that $ab=ba^3$ and show that $ba \neq ab^3$.
I got the idea that this group could give an counterexample based on how asymmetric the cycle graph looks, but I would expect that it's possible to see that the map gives above doesn't induce an automorphism, based on the asymmetric group presentation itself.
Working with the group presentations though, the presentation of the quaternion group is also seemingly asymmetric, and yet if I map $i,j$ to any $x,y$, where $x,y$ are both of order $4$, I will get an automorphism (concretely, $i \mapsto j$ and $j \mapsto i$ induces an automorphism).
More exactly, using the presentation $\langle\,a,b\mid a^4=1\,, b^4=1 \,, a^2=1 \,, b^2=1 \,, bab=a^{-1} \rangle$ of quaternions I can see that from mapping $bab=a^{-1}$ I get $aba$ and $b^{-1}$ and those are in fact equal - as $bab =a^{-1}$ iff $aba = b^{-1}$.
Still, it's not really obvious to me exactly which equations will cause problems (like $ab=ba^3$), causing $f$ not to induce an automorphism, and exactly which won't.
a - you are right, you need only check that $f$ induces a homomorphism. How you might check this depends on how $G$ is represented, so you may need more information for a useful answer - for example if $G=\langle a,b \mid R\rangle$ is a group presentation you need only check that the relations in $R$ map to the trivial element in $G$ (sadly this problem is unsolvable in general).
b - the group you have given is a counter example. The group is $C_4\rtimes C_4$ with the first copy of $C_4$ generated by $a$ and the second by $b$ and $b^{-1}ab=a^3=a^{-1}$. So $ba=b^{-3}a=b^{-3}ab^3b=a^3b\ne ab^3$