In Beachy and Blair's 3rd edition of Abstract Algebra the following problem is given:
Let $H$ and $K$ be subgroups of the group $G$, and let $S$ be the set of left cosets of $K$. Define a group action of $H$ on $S$ by $a \cdot (xH) = ax K$ for $a \in H$ and $x \in G$. By considering the orbit of $K$ under this action, show that $|HK| = \frac{|H||K|}{H \cap K}$
First, I think there is a typo, the action of $H$ on $K$-cosets should be $a \cdot (xK) = axK$. Second, I can't seem to obtain the desired outcome from studying the orbit of $K$. I get: $$ \mathcal{O}(K) = \{ hK \ | \ h \in H \} $$ for $hK = h'K$ we need $h'h^{-1} \in K$ where $h,h' \in H$ hence $h'h \in H \cap K$ which gives $h(H \cap K) = h'(H \cap K)$. Thus, it seems to me there are $|H \cap K|$-representatives for each $K$ coset, in short, $$ |\mathcal{O}(K) = \{ hK \ | \ h \in H \}| = \frac{|H|}{|H \cap K|}.$$ On the other hand, the stabilizer of $K$ is $$ G_K = \{ h \in H \ | \ hK = K \} = H \cap K $$ So, the orbit stabilizer theorem gives $$ |H| = |\mathcal{O}(K)||G_K| = \frac{|H|}{|H \cap K|}|H \cap K| = |H|. $$ True, but, no use to the desired outcome describing the size of $HK$.
Question: how do we show $|HK| = \frac{|H||K|}{H \cap K}$ by applying the theory of group action to the given group action of $H$ on $G/K$?
My approach is not productive. Of course, I know this problem can be solved without group actions (see Dummit and Foote page 93, Prop 13 for example). If my orbit or stabilizer is bogus, please let me know the error of my ways. Thanks in advance for your insights.
The orbit of $K\in G/K$ under the action of left multiplication by $H$ will be $HK/K$. Even though the set $HK$ is not a group, it is a union of left cosets of $K$ so the notation makes sense. Since the left cosets $hK$ (with $h\in H$) partition $HK$, we have $|HK/K|=|HK|/|K|$. You've already shown that this equals $|H|/|H\cap K|$, so you're good.
If you're not working with finite groups, we can still make sense of $|HK/K|=|H/H\cap K|$ as a true equality. In fact if $K$ is infinite and $H$ finite, then this is a stronger claim than the original one stating $|HK|=|H||K|/|H\cap K|$ (which will be trivial in that case).
Alternatively, we can have $H\times K$ act on $HK$ by $(h,k)x=hxk^{-1}$. The action is transitive and the stabilizer of $e$ will be $\cong H\cap K$, which yields the result as well.