I would like to prove that if a non-comutative group $G$ has a subgroup $H$ whose index is equal to $3$ or $4$, then $G$ is not simple.
In order to do, I took a group action $$\phi : G \times G/H \ni (g, g'H) \longmapsto (gg')H \in G/H.$$ $\phi$ induces a group homomoriphism $\varphi : G \rightarrow S_n$($n=|G/H|$), where $S_n$ is the symmetric group on a set of $n$ elements.
If it is proved that the inverse image $\varphi^{-1}({\rm Im}\,\varphi \cap A_n)$($A_n$ is alternating group) is nontrivial in $G$, then original problem is solved. However, I could not prove it. Would you have any hints?
Because you are now assuming $G$ is nonabelian, if $H$ has index $3$ or $4$ then it is not trivial.
Consider first the case of index $3$.
We note that if $\phi(G)$ is not contained in $A_3$, then $S_3/A_3 = \phi(G)A_3/A_3 \cong \phi(G)/(\phi(G)\cap A_3))$, which would show that $\phi(G)\cap A_3$ is of index $2$ in $\phi(G)$ and so lifts to normal subgroup of index $2$, as desired.
Consider the action of $G$ on the left cosets of $H$. If $h\in H$, then $h$ fixes the coset $H$. If there is an $h\in H$ and a $g\in G$ such that $hgH\neq gH$, then the action of $h$ on the left cosets is not trivial and fixes one of the three cosets, so $\phi(h)$ is a transposition and $\phi(G)$ is not contained in $A_3$, so we are done: we obtain a normal subgroup of index $2$, which is not trivial since $G$ has a subgroup of index $3$.
If for every $h\in H$ and every $g\in G$ we have $hgH = gH$, then $g^{-1}hg\in H$ for all $g\in G$ and all $h\in H$, so $H$ itself is normal and of index $3$. Since $G$ is nonabelian, $H$ is nontrivial, so $H$ itself is a nontrivial proper normal subgroup of $G$, which shows $G$ is not simple.
For $n=4$ we again need to ask whether $\phi(G)$ is contained in $A_4$; if it is not contained in $A_4$ then the pullback of $\phi(G)\cap A_4$ will be of index $2$ and normal. On the other hand, if the image is contained in $A_4$, then we can look at the intersection with $V=\{e,(12)(34),(13)(24),(14)(23)\}$, which is normal of index $3$ in $A_4$. Again, note that if $h\in H$, then $h$ fixes the coset $H$; if the action of $h$ is nontrivial, then it must fall outside of $V$, so $\phi(G)\cap V$ will be a proper subgroup of $\phi(G)$ of index $3$. If every $h\in H$ fixes every left coset of $H$ in $G$, then $H$ is normal as above, so $H$ itself gives a proper nontrivial subgroup of $G$, as desired.