Let $H$ be a normal subgroup of $G$ and let $a$ belong to $G$. If the element $aH$ has order $3$ in the group $G/H$ and $|H| = 10$, what are the possibilities for the order of $a$?
Doubt -
1) $|aH|$ has order $10$ in $G$ . I am not able to get a clear picture of what is meant by order $3$ in $G/H$.
I think $|aH|$ can means two things . It can represent size of the the set. And it can also represent order of element $aH$ in $G/H$.
2) $|G|= |aH| *|H|$. Is this right?
Since order of $aH$ must divides $|a|$, $|a|$ must be multiple of $3$.
Also $a^3H = H$, which implies $a^3$ lies in $H$.
By Lagrange's theorem, $|a^3|$ can be $1,2,5$ or $10$.
$|a| = |a^3| * GCD(|a|,3)$
Is this right till here?
How to proceed after this?
Any hint or suggestion will be much appreciated.
"$G/H$" means to take the group $G$ and parititon it into pieces congruent to $H$. This means more than "piece of size $H$", each piece is a translated copy of each other piece. One piece is $eH = \{e h \mid h \in H\}$. The other pieces are constructed by left-multiplying $H$ by elements of $G$. Each $gH$ is a congruent copy of $H$ and the collection of $\{gH \mid g \in G\}$ is all of the congruent pieces, usually with many repetitions. Suppose $g \in H$, then since $H$ is a group (closed under multiplication), for each $h \in H$, $g h \in H$, so $gH = H = eH$, that is, the congruent copy of $H$ made by (left-)multiplying by an element of $H$ is the same (coset) as $eH$.
1) "$|aH|$" does not appear in the problem statement so you are choosing to confuse yourself by using it. Since $aH$ as written is not an element of a group, the vertical bars mean "size as a set". To clarify this to any reader (including yourself), recall that the purpose of notation is to clarify your meaning, so establish your semantics before using the notation: "In $G/H$, the coset $aH$ has order $|aH| = 3$ ..." or "The set has cardinality $|aH| = 10$ ...".
"$aH$ has order $3$ in the group $G/H$" means that $aH$, $(aH)^2$, and $(aH)^3$ are all different and the last one is $eH$. That is, we have listed three different congruent copies of $H$, the last one of which is $eH$. However, we have a convenience, since $H$ is normal in $G$ (necessary condition for the quotient $G/H$ to be a group), $$ (aH)^2 = aHaH = aaHH = a^2H \text{,} $$ where we have used that $H$ is normal to swap $Ha$ to $aH$ and then that $H$ is a group to observe that the product of any two elements of $H$ is an element of $H$ (i.e., $HH = H$). Applying the same argument to $(aH)^3 = a^3H = eH$, it must be the case that $a,a^2 \not\in H$ and $a^3 \in H$. So we know that $G$ contains at least three distinct cosets (congruent copies) of $H$. There could be more, but powers of $a$ cannot tell us about them.
2) We would only have $|G| = |H| \cdot |aH|$ if the powers of $aH$ included all the cosets of $H$ in $G$, but we don't have enough information to claim this. The powers of $aH$ give us three cosets, but there could be more. Another way to say this is that we do not know $G/H$ is cyclic, generated by $aH$. Consider $\mathbb{Z}/8\mathbb{Z}$ under addition. This group is cyclic (generated by each odd element), but if we pick $a = 4$, $a$ has order $2$ so does not generate this group. Our choice of $a$ in the problem statement could be equally under-informative.
You would be able to apply Lagrange's theorem to the order of $a$ as an element of the group $H$ if $a$ were an element of $H$. But it isn't; we already know $a$ and $a^2$ are not elements of $H$, so Lagrange's theorem does not apply. Lagrange's theorem tells you that $|H|$ divides $|G|$, so $|G|$ is a multiple of $10$. By other means (for instance, the first isomorphism theorem, which you may not have in your toolbox yet), you know the order of $aH$ in $G/H$ divides the order of $a$ in $G$ and the order of $a$ is the cardinality of the cyclic group generated by $a$ (BTW: This last equivalence is the source of the re-use of the vertical bar notation for order of an element. Although "$|\langle a \rangle |$" would be more accurate, no one writes that.) so Lagrange's theorem tells you $3 \mid |G|$. Putting these together, $\mathrm{lcm}(3,10) = 30$ divides the order of $G$. That's as far as Lagrange's theorem speaks to this problem.
Perhaps a better, elementary, way to attack this problem is this. Consider the list of distinct powers of $a$ (in $G$): $$ \underline{e}, a, a^2, \underline{a^3}, a^4, a^5, \underline{a^6}, \dots \text{.} $$ (I've explicitly shown the first seven powers of $a$, but we do not know that these seven are distinct -- we only know the first three are distinct.) We know that $a^3 \in H$, so the underlined elements must be distinct elements of $H$. Since there are only $10$ elements of $H$, the pigeonhole principle tells us there can be only up to $10$ underlined powers of $a$ on this list and there could easily be fewer.