Let $G$ be a group.
Let $N$ be a normal subgroup of $G$.
Let $H$ be a subgroup of $G$ such that $N\nsubseteq H$.
Define a relation $\sim$ on $H$ as follows:
$a\sim b$ if and only if $a^{-1}b\in N$.
Then, this relation is an equivalence relation on $H$.
Let $\overline{a}:=\{x\in H\mid a\sim x\}$ for $a\in H$.
Let $H{'}:=\{\overline{a}\mid a\in H\}$.
Define a product on $H'$ as follows:
$\overline{a}\overline{b}:=\overline{ab}$.
Then, this product is well-defined:
If $a_1\sim a_2$ and $b_1\sim b_2$, then $(a_1b_1)^{-1}(a_2b_2)=b_{1}^{-1}a_{1}^{-1}a_2b_2\in b_1Nb_{1}^{-1}=N$.
So, $a_1b_1\sim a_2b_2$.
$H'$ is a group under this product:
Obviously, the associativity law holds.
$\overline{e}\in H{'}$ is the unit element of $H'$.
If $\overline{a}\in H{'}$, then $\overline{a^{-1}}\in H{'}$ since $a^{-1}\in H$.
And $\overline{a^{-1}}$ is the inverse of $\overline{a}$.
The authors of algebra books don't write this group $H{'}$ in their books.
Why?
If I understand your question right, this group $H'$ that you mention is interesting to you, and you are wondering why this group is not used in algebra textbooks.
It actually is used. Your $H'$ is nothing more than the quotient group $HN/N$, where $HN = \{ hn : h \in H, n \in N\}$ is a subgroup of $G$ containing $N$ as a normal subgroup.
As mentioned in the comments, another common expression of what you call $H'$ is the quotient group $H/(H \cap N)$. The natural isomorphism between $HN/N$ and $H/(H \cap N)$ is the second isomorphism theorem.