Let $A=\{2,4,5\}$ and let $H=\{F\in S_q \mid F(A)=A\}$
(a) Show that $H$ is a subgroup of $(S_q,\circ)$
(b) What is the order of $H$, $|H|=?$
Not quite finding out how to do the first section, I am trying to apply that for all $F, G^{-1} \in H$ we have $F \circ G^{-1}\in H$ their composition but I dont know what exactly $G^{-1}$ looks like.
(a) First, you shouldn't assume $g^{-1}\in H$ --- that's not how subgroups work. Instead, you should assume $f,g\in H$ and show $f\circ g^{-1}\in H$.
I think it's conceptually simpler to break it into two steps: (1) show that $f\circ g \in H$, and (2) show that $g^{-1}\in H$.
Hint: if $f(A)=A$ and $g(A)=A$, how can you show that $(f\circ g)(A)=A$?
(b) For this, let's do an example to see what's going on. Say $q=7$. Then a permutation $f\in S_7$ would have to preserve the set $A=\{2,4,5\}$, but then it would also have to preserve the complement $A^c=\{1,3,6,7\}$! (Think about why --- maybe try to come up with a few examples of permutations that are in $H$, and permutations that are not in $H$.)
Therefore, $f$ is acting as two different permutations: one on $\{2,4,5\}$, and one on $\{1,3,6,7\}$. How many such $f$ are there?