G be a finite group and H be a non trivial proper subgroup of index 3, then G is not simple.

Question

If the index of a subgroup is the smallest prime dividing the order of group then that subgroup is normal, with this, I am done with the case order of group is odd. How to proceed when order is even.

Answer 1

$H < G$, $[G : H] = 3.$ This defines action of $G$ on cosets $G/H$ — so there's a homomorpism $G \to S_3$ to symmetric group on 3 elements. If $|G| > 6$, then kernel is nontrivial, so $G$ is not simple. If $|G| \leq 6$, then (as there's index 3 subgroup) $|G| = 1, 3, 6$. First two possibilities are ruled out by existence of nontrivial index 3 subgroup, and there are 2 groups of order 6 (let's leave this as exercise) neither of which are simple.

Answer 2

In general: if $G$ is a finite non-abelian simple group and $H$ is a subgroup, then $|G:H| \geq 5$.

Proof (Sketch) Assume $|G:H| \leq 4$. Let $G$ act by left multiplication on the left cosets of $H$. The kernel of this action is $core_G(H)=1$, since $G$ is simple. Hence $G$ can homomorphically be embedded in $S_{|G:H|}$. Since $S_4$ is solvable and $G$ is simple, it follows that $G=1$, a contradiction.

Observe that $A_5$ is the first example of a non-abelian simple group and that $|A_5:A_4|=5$.