Prove that the automorphism group $\operatorname{Aut}(C_n)$ of a cyclic group of order $n$ is isomorphic to $(\mathbb{Z}/n\mathbb{Z})^*$
I know that the automorphism group consists of all the homomorphisms from the group to the group. And I know that somehow I need to use the fact that the group is cyclic and has a generator. But I can't quite put the puzzle together.
Let $\langle g\rangle $ be the generator, it exists $m \in \mathbb{N}$ s.t $g^m=1$ so an automorphism must do $f(1)=1=f(g^m)=f(g)...f(g)=(f(g))^m=1$ does that mean $f(g)$ is also a generator ?
In order to have an isomorphism, I need to consider a homomorphism. Let $\varphi:\operatorname{Aut}(C_n) \rightarrow \mathbb{Z_n}$ with $\varphi(f)=$ ?
That's how far I go, automorphism are new to me and the whole exercise confuses me a little bit.
This is one place where I think it is very helpful to first ponder the problem in slightly more generality. Fix a generator for $C_n$ once and for all. I'll call it $g$ to meet your notation, so $C_n = \langle g \rangle$ throughout.
First, prove that every homomorphism $f: C_n \to C_n$ is uniquely determined by the value $f(g)$. I think you know how to do this. You just need to prove that given homs $f,h$ we have $f=h$ if and only if $f(g)=h(g)$. This is nothing more than the definition of cyclic group.
Next, prove that every homomorphism $f: C_n \to C_n$ can be associated to a unique integer $m$ with $0 \leq m \leq n-1$. The association is: given a hom $f$, we know that $f(g) \in C_n$, but $C_n$ is generated by $g$. Thus $f(g)$ is expressible as $g^m$ for some unique power $m$ with $0 \leq m \leq n-1$. Let's call this $m_f$ instead of $m$ to denote its dependence on $f$. In short, $f(g)=g^{m_f}$ where $m_f$ has been chosen "correctly."
You should be able to check that the association $f \mapsto m_f$ defines a perfectly good function from the set of all homs $f: C_n \to C_n$ into $\mathbb{Z}_n$ (as sets only for the moment).
Here's the first of two crucial moves: prove that given two homs $C_n \stackrel{f}{\rightarrow}C_n\stackrel{h}{\rightarrow} C_n$ we have $m_{h\circ f}=m_h m_f \bmod n$. That is, our association $f \mapsto m_f$ respects composition in a multiplicative way mod $n$. This is easier than it looks.
Here's the final bit. Prove that $f: C_n \to C_n$ is an automorphism if and only if $m_f$ has a multiplicative inverse mod $n$. To show one direction, suppose $f$ is an automorphism. Then as $f$ is surjective there must be an $a \in C_n$ with $f(a)=g$. This $a$, however, is expressible as $a=g^k$ for some integer $k$, since $g$ generates $C_n$. Then $$ g=f(a)=f(g^k)=f(g)^k=(g^{m_f})^k = g^{m_fk}. $$ Hence $g^{m_fk}=g^1$ so you now have that $m_fk \equiv 1 \bmod n$, which is what we wanted to show. You can fill in the rest.
Note that the association $f \mapsto m_f$ gives your isomorphism $\operatorname{Aut}(C_n) \to \mathbb{Z}_n^*$ and it is now clear why the operation must be multiplication mod $n$ (because the operation in the automorphism group is composition).