Suppose $G$ is a finite group and $N \triangleleft G$. Let’s define the relative commuting fraction as $$cf(G, N) := \frac{|\{(g, h) \in G \times H| [g, h] = e\}|}{|G||H|}$$
Does there exist such $\epsilon > 0$, that for all $G$ and $H$ if $cf(G, N) > 1 - \epsilon$, then $N \leq Z(G)$?
For the particular case when $N = G$ we have the following fact:
Erdos-Turan theorem
If $cf(G, G) > \frac{5}{8}$, then $G$ is abelian.
However, the method that is used to prove it can not be extended onto this general case…
It may still be useful to get this inequality:
$$cf(G, N) \leq \frac{|Z(G) \cap N|}{|N|} + \frac{1}{2}$$
$$cf(G, N) = \frac{1}{|G||H|}(|G||Z(G)\cap N| + \Sigma_{g \in N} C_G(g)) \leq \frac{1}{|G||H|}(|G||Z(G)\cap N| + \frac{|N||G|}{2}) = \frac{|Z(G) \cap N|}{|N|} + \frac{1}{2}$$
But I do not know how to proceed further...
You've done most the work finding the above bound, but you made one subtle error.
The sum should be over $g\in N\setminus(Z(G)\cap N)$.
If you follow this through you get $$cf(G,N)\le\frac{|Z(G)\cap N|}{2|N|}+\frac{1}{2}$$
So if $cf(g,N)>3/4$ then $\frac{|Z(G)\cap N|}{|N|}>\frac{1}{2}$ and $Z(G)\cap N=N$