Let $G= \left\{ \begin{pmatrix} a & b \\ 0 & 1 \end{pmatrix} \text{ with } a \text{ in } \{1,3,9\} \subset F^{*}_{13} \text{ and } b \text{ in } F_{13} \right\}$, where $F^{*}_{13}$ is the group modulo $13$ under multiplication and $F_{13}$ the group modulo $13$ under addition. Show it has seven conjugacy classes with respectively $1,3,3,3,3,13,13$ elements and prove it has exactly three normal subgroups.
I started this problem by looking at the conjugacy classes, that is for $g_{1}g_{2}g^{-1}_{1}=g_{3}$ for some $g_{1} \in G$.
Thus I let $g_{1} = \begin{pmatrix} a & b \\ 0 & 1 \end{pmatrix}$ and $g_{2} = \begin{pmatrix} c & d \\ 0 & 1 \end{pmatrix},$ then $$g_{1}g_{2}g^{-1}_{1}=\begin{pmatrix} c & -bc+b+ad \\ 0 & 1 \end{pmatrix}.$$
Now per conjugacy class this needs to be an element of $G$. So I guess I have to check the order per assigned value of $c, a, b$ and $d$. Correct? Now $c$ can either be $1,3,9$, thus let's start with
$c=1 \rightarrow \begin{pmatrix} 1 & ad \\ 0 & 1 \end{pmatrix}$, now $a$ can be one of three things but $d$ can be any of $13$. So I would argue this conjugacy class already has $3 \cdot 13 = 39$ elements. Obviously I go wrong, but where and why?
I don't even know where to begin to argue it has exactly three normal subgroups.
The conjugacy class containing element $g \in G$ is defined as the set of elements $g'$ such that there exists $h \in G$ satisfying $g' = hgh^{-1}$. The thing you've done wrong is allow $g_2$ to vary within the same conjugacy class; this is not always allowed.
What you want to do here is pick a fixed $g_2$, and then find all possible elements of the group of the form $g_1 g_2 g_1^{-1}$.
The conjugacy class with one element is simply the identity matrix. Also note that the top-left element of the matrix $g_1 g_2 g_1^{-1}$ is the same as that of $g_2$, so every element in the same conjugacy class has the same top-left element. Note that if $c \neq 1$, the top-right entry is $b(c-1) + ad \in F_{13}$. Since $13$ is prime and $c-1 \neq 0$, we simply let $b$ vary over $F_{13}$, and note that $b(c-1)$ must take every value modulo 13 since multiples of a nonzero number in a prime mod take every value in that mod.
So we can obtain every possible top right entry, and so the set of elements in $G$ with a top-left entry of $3$ form a conjugacy class, and the set of elements with a top-left entry of $9$ form another conjugacy class. This gives us the two classes of size $13$.
It remains to divide the twelve elements remaining into conjugacy classes. The top-right entry is now $ad$, for fixed $d \neq 0$ and $a = 1, 3, 9$. Clearly this yields three distinct values, and so each element remaining is in a conjugacy class of size $3$. This completes the first part.
To find normal subgroups, note that the entire $G$ is clearly a normal subgroup, as is the trivial group only containing the identity matrix $I$. The third normal subgroup is the set of elements with $a=1$; note that this is a subgroup as the top-left element of $g_1 g_2$ is $1$ for any $g_1, g_2$ in this subgroup, and note that it is normal because the conjugate of any element in this subgroup still has top-left element $1$ and therefore is contained in the subgroup.
We now need to prove there are no other normal subgroups. Let the normal subgroup with all elements $a=1$ be called $A$. Now any non-identity element in $A$ can be repeatedly multiplied to itself to yield all elements of $A$ (try proving this), so any normal subgroup containing any non-identity element in $A$ must contain $A$ itself. Also note that for any element $g$ not in $A$, the elements $g$, $g^2$ and $g^3$ all have different top-left entry (again try proving this). Since a normal subgroup must contain the conjugacy class of every element in it, we know that the two conjugacy classes of size $13$ are in this subgroup. It now remains to show that some non-identity element of $A$ is in this subgroup. Try finding an appropriate $g$ with $a=3$ or $9$ such that $g^3$ is not the identity (I'll leave this as an exercise). From here, since this normal subgroup contains a non-identity element of $A$, it must contain the entire $A$ (we derived this earlier), and so the subgroup is the entirety of $G$.