I am trying to understand how a group homomorphism mapping generators to generators can be shown to be an isomorphism.
The example I have in mind is $S_3$ and $D_3$.
The group presentations I have in mind are: \begin{align*} S_3 &= \langle (123), (12) \mid (123)^3 = 1, \; (12)^2 = 1, \; (12)(123) = (123)^2 (12) \rangle \\ D_3 &= \langle r, s \mid r^3 = 1, \; s^2 = 1, \; sr = r^{-1} s \rangle. \end{align*} The map $f: S_3 \to D_3$ would send $(123)$ to $r$ and $(12)$ to $s$. This preserve the order of the generators, and the homomorphism is now uniquely determined. I don't know if that is enough since the groups have the same number of elements and no element is duplicated. I thought it may be enough to define this map and check that the generators satisfy the defining relations. The elements $r$ and $(123)$ have order $3$ and the elements $(12)$ and $s$ have order $2$, so that's the first two relations. In $D_3$, $r^{-1} = r^2$, so I can rewrite the third relation for $D_3$ as $sr = r^2 s$, which is exactly the same as the relation for $S_3$, replacing $s$ by $(12)$ and $r$ by $(123)$.
Is this enough to establish an isomorphism?
In general, just assigning values to the generators is not sufficient; you need to make sure you have a homomorphism. Verifying it is a homomorphism from first prinicples is not a trivial matter if all you know is what happens to the generators! Of course, knowing that the image of $x$ has the same order as $x$ is necessary for the function to define an isomorphism, but it is not sufficient by a long shot.
For example, consider $D_3$ and $D_4$. $$\begin{align*} D_3 &= \langle r,s\mid r^2=s^2=(rs)^3 = 1\rangle,\\ D_4 &= \langle x,y\mid x^2 = y^2 = (xy)^4 =1\rangle. \end{align*}$$ Now, $r$ and $x$ have the same order; $s$ and $y$ have the same order. Is there a homomorphism such that $r\mapsto x$ and $s\mapsto y$? No. Checking that the order of the image equals the order of the original element is not enough.
The above example uses that $D_n$ can be defined as $$D_n = \langle a,b\mid a^2=b^2=(ab)^n=1\rangle.$$ See below for a proof that this presentation gives a group isomorphic to $$D_n = \langle r,s\mid r^n=s^2=1, sr=r^{-1}s\rangle.$$
(If you somehow know you have a homomorphism and you know the image contains a set of generators of the codomain, then you know the homomorphism is surjective. But you don't know it is one-to-one, so you don't know if it is an isomorphism. You would need to show that the kernel is trivial. And that is also not always easy to do from first principles.)
In this situation, what you have are groups given by generators and relations. That suggests that what you want to do is use van Dyck's Theorem:
Here you have the following presentation. $$D_3 = \langle r,s\mid r^3, s^2, srs^{-1}r\rangle.$$
To apply van Dyck's Theorem to obtain a map from $D_3$ to $S_3$, you select values for $r$ and $s$, and verify that those values satisfy the equations $r^3=e$, $s^2=e$, $srs^{-1}r=e$.
For example, if we assign $(123)$ to $r$ and $(12)$ to $s$, we need to check that $(123)^3=e$, $(12)^2 = e$, and $(12)(123)(12)^{-1}(123) = e$. They are all true, hence by van Dyck's Theorem there exists a unique group homomorphism $\varphi\colon D_3\to S_3$ such that $\varphi(r)=(123)$ and $\varphi(s)=(12)$. Because we know that $(123)$ and $(12)$ generate $S_3$, we know this map is a surjection.
If we also happen to know the orders of the groups, this may be enough to decide that this is a bijection; but that requires more than just knowing you can define the map. Here we know that $D_3$ has six elements as well, so this map must be a bijection. Since van Dyck's Theorem guarantees it is a homomorphism, this proves it is in fact an isomorphism.
Note that while $D_3$ is usually defined using the presentation you give, $S_3$ is not. You would need to prove that $$\langle x,y\mid x^3, y^2, yxy^{-1}x^{-2}\rangle$$ is actually a presentation for $S_3$. That would itself require a use of van Dyck's Theorem: let $G$ be the group given by the presentation above. Then the assignment $x\mapsto (123)$ and $y\mapsto (12)$ is such that $(123)^3 = (12)^2 = (12)(123)(12)^{-1}(123)^{-2}=e$, so by van Dyck's Theorem we get a homomorphism $\varphi\colon G\to S_3$; because the image contains a generating set, the map is surjective. But is it an isomorphism? That requires verifying that it is one-to-one.
One way to do that is to get at least a bound for the size of $G$. Using the fact that $x^3=e$ and $y^2=e$, one can show that any product of powers of $x$ and $y$ can be rewritten a product of powers where the exponents of $x$ are between $0$ and $2$ and the exponents of $y$ are between $0$ and $1$. Then using $yx=x^2y$ we can show that any such expression can be rewritten in the form $x^iy^j$, where $0\leq i\leq 2$ and $0\leq j\leq 1$. We are not asserting (yet) that different expressions yield different elements. But this shows that $G$ has at most six elements.
Since we already know that $G$ has at least six elements, given that there is a surjection $G\to S_3$, this proves that $G$ has exactly six elements, and therefore the surjection $\varphi\colon G\to S_3$ given above must be a bijection. Since van Dyck's Theorem guarantees that it is a homomorphism, we get that $G$ is an isomorphism.
By the way, let us use van Dyck's Theorem to prove that the two presentations of $D_n$ actually define the same group (up to isomorphism). that is, that $$G=\langle r,s\mid r^n=1, s^2=1, sr=r^{-1}s\rangle \cong \langle a,b\mid a^2=1, b^2=1, (ab)^n=1\rangle=H.$$
Consider the map $f\colon\{r,s\}\to H$ given by sending $f(r)=ab$ and $f(s)=b$. We need to verify that $f(r)^n = f(s)^2 =1$, and that $f(s)f(r) = f(r)^{-1}f(s)$. Indeed, we have $$\begin{align*} f(r)^n &= (ab)^n = 1,\\ f(s)^2 &= b^2 = 1,\\ f(s)f(r) &= bab = (ba)b = (b^{-1}a^{-1})b = (ab)^{-1}b = f(r)^{-1}f(s). \end{align*}$$ By van Dyck's Theorem, $f$ extends to a homomorphism $\varphi\colon G\to H$ such that $\varphi(r)=ab$, $\varphi(s)=b$.
Now define $g\colon \{a,b\}\to G$ by $g(a) = rs$ and $g(b)=s$. We need to verify that $g(a)^2 = g(b)^2 = (g(a)g(b))^n=1$ in $G$. Indeed, $$\begin{align*} g(a)^2 &= (rs)^2 = rsrs = r(r^{-1}s)s = s^2 = 1,\\ g(b)^2 &= s^2 = 1,\\ (g(a)g(b))^n &= (rss)^n =r^n = 1. \end{align*}$$ By van Dyck's Theorem, we obtain a homomorphism $\psi\colon H\to G$ such that $\psi(a)=rs$ and $\psi(b)=s$.
Now note that $\psi\circ \varphi\colon G\to G$ and $\varphi\circ\psi\colon H\to H$ are certainly homomorphisms, and they restrict to the identity on generating sets: $$\begin{align*} \psi(\varphi(r)) &= \psi(ab)=\psi(a)\psi(b) = rss = r,\\ \psi(\varphi(s)) &= \psi(b) = s,\\ \varphi(\psi(a)) &= \varphi(rs) = \varphi(r)\varphi(s) = abb=a,\\ \varphi(\psi(b)) &= \varphi(s) = b. \end{align*}$$ Therefore, $\psi\circ\varphi=\mathrm{id}_G$ and $\varphi\circ\psi=\mathrm{id}_H$, so $G\cong H$.