Proposition. Let $G$ be a group, $H \le G$, and $f \colon G \times G \rightarrow G$ the group operation. Call $\complement_GH:=G \setminus H$. Then:
$f(H \times \complement_GH) \subseteq \complement_GH$
$f(\complement_GH \times H) \subseteq \complement_GH$
$f(\complement_GH \times \complement_GH)=\begin{cases}\emptyset&\text{if }H=G\\H&\text{if }[G:H]=2\\G&\text{otherwise}\end{cases}$
Proof.
- Let $h \in H,g \in \complement_GH$; by contrapositive, $hg \in H ⇒g=h^{-1}hg \in H$: contradiction; then, $hg \in \complement_GH$.
- Same as 1.
- See the answer here.
$\blacksquare$
I thought to use this Proposition to prove the well known:
Corollary. $[G:H]=2 \Rightarrow H \unlhd G$.
Proof. Take any $h∈H$; if $g∈H$, then $g^{-1}hg∈H$ by definition of subgroup; if $g∈\complement_GH$, then (see 2) $g^{-1}h∈\complement_GH$ and finally (see 3, case in the middle) $g^{-1}hg \in H$. So $g^{-1}hg \in H$ for all $g \in G$; but $h$ is arbitrary in $H$, then $H \unlhd G$. $\blacksquare$
This made me raise the following point (and final question). Suppose that $(G,H)$ is such that $[G:H]>2$ and:
$$\exists \bar h \in H : \lbrace \bar hg \mid g \in \complement_GH \rbrace = \complement_GH \tag 1$$
As per Prop's item 3, bottom case, we'd have then $\lbrace g^{-1}\bar hg \mid g \in \complement_GH \rbrace=G \not \subseteq H$, so that $H \not \unlhd G$.
Can (1) be seen as a virtual (if not practical) "non-normality test" for $H$ in $G$?
There is an error at the last step of your reasoning:
It does not follow that $\{g^{-1}\bar{h}g|g\in \complement_GH\}=G$, only that $\{g^{-1}\bar{h}g|g\in \complement_GH\}\subseteq G$. As $g^{-1}\bar{h}$ depends on $g$ we are not looking at the whole of $f(\complement_GH\times\complement_GH)$.
In fact for any $(G,H)$ with $H\le G$ and any $h\in H$ we have $\{hg|g\in\complement_GH\}=\complement_GH$.
Immediately $hg\in\complement_GH$ and for any $y\in\complement_GH$ we have $h^{-1}y\in\complement_GH$ so $y=hh^{-1}y\in\{hg|g\in\complement_GH\}$.