Surely, $S_3 = \langle(1,2), (1,2,3)\rangle$
And Put $\alpha = (1,2)$ and $\beta = (1,2,3)$ for homomorphism, $\phi : S_3 \to S_3$
Say $\phi(\alpha) =f(\in S_3)$ and $\phi(\beta) =r(\in S_3)$
Then, $\vert f \vert $ and $\vert r \vert$ should be divisor of the $2$ and $3$ respectively.
Hence there are 4 cases like the below.
1) $\vert f \vert = 1$ and $\vert r \vert = 1$ (The number of the $\phi$ is $1$)
2) $\vert f \vert = 1$ and $\vert r \vert = 3$ (The number of the $\phi$ is $2$)
3) $\vert f \vert = 2$ and $\vert r \vert = 1$(The number of the $\phi$ is $3$)
4) $\vert f \vert = 2$ and $\vert r \vert = 3$(The number of the $\phi$ is $6$)
So my answer is $12$.
But the answer of that was $10$ not $12$.
Which point I was wrong? Any advice would be appreciated.
Just because a group $G$ is generated by elements $g_1$ and $g_2$, this does not mean that, whenever $h_1$ and $h_2$ are elements of $G$ with orders dividing those of $g_1$ and $g_2$, respectively, there is an endomorphism of $G$ mapping $g_i$ to $h_i$.
For example, your case 2) is never attainable: Suppose $\phi(1,2)=1$. Then \begin{align*} \phi(1,3)&=\phi((2,3)^{-1}(1,2)(2,3))\\ &=\phi(2,3)^{-1}\phi(1,2)\phi(2,3)\\ &=\phi(2,3)^{-1}\phi(2,3)\\ &=1 \end{align*}
But $(1,2)$ and $(1,3)$ generate $S_3$, so $\phi=1$, the trivial endomorphism.
So the argument should be a little more refined, using the group structure of $S_3$.
As you already notices, $|\phi(a)|$ always divides $|a|$. Let us look at a few cases:
So suppose $\phi(1,2)\neq 1$. Then $\phi(1,2)$ is a $2$-cycle, say $\phi(1,2)=(a,b)$. We have two further possibilities:
$|\phi(1,2,3)|=1$:
In this case, the image of $\phi$ will be contained in $\langle \phi(1,2),\phi(1,2,3)\rangle=\left\{(a,b),1\right\}$, which is the group with two elements. In this case, $\phi$ will just be a realization of the "parity" homomorphism: $$\phi(x)=\begin{cases} (a,b)&\text{, if }x\text{ is odd}\\ 1&\text{ if }x\text{ is even} \end{cases}$$ For these cases, $\phi$ is completely determined by the choice of $a$ and $b$ (and the order does not matter): there are $3\times 2/2=3$ choices for $a$ and $b$ in this case.
$|\phi(1,2,3)|=3$.
So up to changing the order of the $a$ and $b$ (because $(a,b)=(b,a)$), we can write $\phi(1,2,3)=(a,b,c)$.
So we see that $\phi$ is just "swapping $1$ with $a$, $2$ with $b$ and $3$ with $c$".
More formally, let $\tau\in S_3$ be defined as $\tau(a)=1$, $\tau(b)=2$ and $\tau(c)=3$. Then $\phi(x)=\tau x\tau^{-1}$ whenever $x$ is any of the generators $(1,2)$ or $(1,2,3)$. Since the "conjugation by $\tau$", $c_\tau:x\mapsto \tau x\tau^{-1}$ is an endomorphism, we actually have $\phi=c_\tau$ on all of $S_3$.
This means that is $|\phi(1,2)|=2$ and $|\phi(1,2,3)|=3$, then $\phi$ is just conjugating with some element of $S_3$. There are $6$ elements in $S_3$, and all the conjugation endomorphisms (the endomorphisms of the form $c_\tau$ for some $\tau\in S_3$) are different. Since $S_3$ has $6$ elements, there are $6$ such "conjugation morphisms".
In summary, we have - the trivial morphism (if $|\phi(1,2)|=1$); - 3 morphisms if $|\phi(1,2)|=2$ and $|\phi(1,2,3)|=1$; - 6 morphisms if $|\phi(1,2)|=2$ and $|\phi(1,2,3)|=3$;
so in total, 10 endomorphisms of $S_3$.