Let $G \in \mathbf{Grp}$, $H \leq G$, $G/H := \lbrace gH: g \in G \rbrace$. We can then introduce group operation on $G/H$ as $(xH)*(yH) := (x*y)H$, so that $G/H$ becomes a quotient group when $H$ is a normal subgroup.
But why do we only work with quotient groups by normal subgroups? If we introduce the notion of left quotient group in the above manner, how much good properties of a quotient group do we lose?
On
(Sorry, new here, can't comment.)
To expand on Alex's answer, if we demand that $(xh)(yh) \in (xy)H$ for any coset representative $h$, then this is equivalent to asking that $y^{-1}hyh \in H$ for all $h \in H$. But the conjugate $y^{-1}hy$ need not lie in $H$ if $H$ isn't normal, hence the claim that the operation in your "left quotient group" isn't well defined.
On
Added. So, what is the problem? Let's look at the simplest example of a non-normal subgroup. Take $G=S_3$, and $H=\{e,(1,2)\}$. If we compose permutations right to left, the left cosets of $H$ in $G$ are: \begin{align*} eH = (1,2)H &= \{ e, (1,2)\};\\ (1,2,3)H = (1,3)H &= \{(1,2,3), (1,3)\};\\ (1,3,2)H = (2,3)H &= \{(1,3,2), (2,3)\}. \end{align*} If we try multiplying cosets term-by-term, we run into problems. Multiplying by $eH$ is not a problem, but take $(1,2,3)H$ multiplied by itself. The products are: $$\{ (1,2,3)(1,2,3), (1,2,3)(1,3), (1,3)(1,2,3), (1,3)(1,3)\} = \{(1,3,2), (2,3), (1,2), e\}$$ which is not a coset. If we multiply using representatives, as in the original question,we also run into problems: if we multiply $(1,2,3)H\cdot(1,3,2)H$ as $(1,2,3)(1,3,2)H$, we get $eH$. But $(1,2,3)H = (1,3)H$, and $(1,3,2)H = (2,3)H$, and if we multiply them by looking at these alternative representatives/names, we get $(1,3)H\cdot (2,3)H = (1,3)(2,3)H = (1,3,2)H\neq eH$. That is, the multiplication rule depends on the name we give the coset, rather than on what the coset is. This means that the rule is not well-defined.
We can let $G$ act on the set of left cosets of $H$, by $g(xH) = gxH$; this is not a group, but a group action. We can also, separately, put group structures on the set of cosets of $H$ in $G$, since it will be a set, and every nonempty set can be given a group structure (assuming the Axiom of Choice; in fact, equivalent to the Axiom of Choice). But that structure will generally have nothing to do with the operation in $G$.
If we want an operation on the set of cosets, and have the operation in the cosets to be induced by the operation in $G$, then we have:
THEOREM. Let $G$ be a group and $H$ a subgroup. Let $G/H = \{ gH\mid g\in G\}$ be the set of left cosets of $H$ in $G$. The operation $$\cdot \colon G/H \times G/H \to G/H\qquad\text{defined by}\qquad (xH)\cdot(yH) = (xy)H$$ is well defined (that is, if $xH = x'H$ and $yH=y'H$ then $xyH = x'y'H$) if and only if $H$ is normal in $G$.
Proof. Suppose $H$ is normal. Then $gH=Hg$ for every $g\in G$. If $xH=x'H$ and $yH=y'H$, then $$xyH = x(yH) = x(Hy) = x(Hy') = (xH)y' = (x'H)y' = x'(Hy') = x'(y'H) = x'y'H$$ so the operation is well defined.
Conversely, suppose the operation is well defined so that whenever $xH=x'H$ and $yH=y'H$, then $xyH = x'y'H$. We want to show that $g^{-1}Hg\subseteq H$ for all $g\in G$. For each $h\in H$, since $hH = eH$, we have $eHgH = egH=gH$ is equal to $hHgH = hgH$. So $gH = hgH$, hence $H=g^{-1}hgH$, so $g^{-1}hg\in H$ for each $h\in H$. Thus, $g^{-1}Hg\subseteq H$. This holds for any $g\in G$, so $g^{-1}Hg\subseteq H$ for all $g\in G$, hence $H\triangleleft G$, as claimed. QED
And now I'm going to hijack your question a bit and go on a long excursion to a closely related subject which is not necessary to answer your question, but which I think is interesting. (Sorry!)
This is really connected to the notion of a congruence on an algebra (in the sense of universal algebra). A subgroup $H$ defines an equivalence relation on $G$ by $x\sim y$ if and only if $xH = yH$ (it also defines the equivalence relation of right congruence, but ignore that for now). Being an equivalence relation, it corresponds to a subset $\Phi$ of $G\times G$, with $(x,y)\in\Phi$ if and only if $x\sim y$, if and only if $xH=yH$.
But $G\times G$ also has a natural group structure inherited from $G$. Since $\Phi$ is a subset of the group $G\times G$, it makes sense to ask whether $\Phi$ is a subgroup of $G\times G$ or not.
THEOREM. $\Phi$ is a subgroup of $G\times G$ if and only if $H$ is normal in $G$.
Proof. Note that $\Phi$ always contains $(e,e)$ (in fact, the entire diagonal). Assume first that $H$ is normal. If $(x,y)\in \Phi$, then $xH=yH$. Hence, $Hx = Hy$, and taking inverses of each element we get $x^{-1}H = y^{-1}H$, so $(x^{-1},y^{-1})\in\Phi$; thus, $\Phi$ is closed under inverses. If $(x,y),(z,w)\in\Phi$, then $xH=yH$, $zH=wH$, so by what we proved above we have $xzH = ywH$, so $(xz,yw)=(x,y)(z,w)\in\Phi$. Thus, $\Phi$ is closed under products, so $\Phi$ is a subgroup of $G\times G$.
Conversely, suppose that $\Phi$ is a subgroup. If $xH=yH$ and $zH=wH$, then $(x,y),(z,w)\in \Phi$, hence $(x,y)(z,w)=(xz,yw)\in\Phi$; thus, $xzH = ywH$. This proves that the operation $xH\cdot zH \mapsto xzH$ is well-defined, so by the theorem above $H$ is a normal subgroup of $G$. QED
Okay, how about this? Can we define some equivalence relation $\sim$ on a group $G$ so that we can multiply equivalence classes through representatives, have it be well-defined, but not have $\sim$ "come from" a normal subgroup?
That is: suppose $\sim$ is an equivalence relation on the group $G$ (any equivalence relation; all we require is that it be reflexive, symmetric, and transitive on the underlying set of $G$), and let $[x]$ denote the equivalence class of $x$ for each $x$ in $G$. We want to define a multiplication on the quotient set $G/\sim$ by $[x][y] = [xy]$. When is this well defined?
THEOREM. Let $G$ be a group, and let $\sim$ be an equivalence relation on $G$. Then the multiplication on $G/\sim$ defined by $[x][y]=[xy]$ is well defined if and only if $\sim$ is a subgroup of $G\times G$.
Proof. If $\sim$ is a subgroup of $G\times G$ and $[x]=[z]$, $[y]=[w]$, then $(x,z),(y,w)\in \sim$, so $(x,z)(y,w) = (xy,zw)\in \sim$, hence $[xy]=[zw]$, so multiplication is well defined.
Conversely, suppose the multiplication is well defined. Since $\sim$ is an equivalence relation, $(e,e)\in\sim$. Also, if $(x,z), (y,w)\in\sim$, then $[x]=[z]$, $[y]=[w]$, so $[x][y]=[xy]$ equals $[z][w]=[zw]$, so $(xy,zw)=(x,y)(z,w)\in\sim$. Thus, $\sim$ is a submonoid of $G\times G$. It only remains to show that $\sim$ is closed under inverses. Suppose that $(x,y)\in\sim$. Multiplying by $(x^{-1},x^{-1})$ on the left and $(y^{-1},y^{-1})$ on the right (both elements of $\sim$) we have $(y^{-1},x^{-1})\in\sim$. Since $\sim$ is an equivalence relation, it is reflexive, so $(x^{-1},y^{-1})\in\sim$. Thus, $\sim$ is a subgroup of $G\times G$, as claimed. QED
Okay, so we can define the multiplication on the quotient set if and only if the equivalence relation is a subgroup of $G\times G$. Do all such equivalence relations "come" from subgroups? Yes:
THEOREM. Let $\sim$ be an equivalence relation on $G$ that is a subgroup of $G\times G$, and let $H=\{g\in G\mid g\sim e\}$. Then $H$ is a normal subgroup of $G$, and $x\sim y$ if and only if $xH=yH$.
Proof. We can identify $G$ with the subgroup $G\times\{e\}$ of $G\times G$; then $H$ corresponds to the intersection of the subgroup $\sim$ and the subgroup $G\times\{e\}$, hence is a subgroup of $G$.
To see that $\sim$ is in fact left congruence modulo $H$, simply note that $xH=yH$ if and only if $y^{-1}x\in H$, if and only if $(y^{-1}x,e)\in \sim$, if and only if $(y,y)(y^{-1}x,e)\in\sim$, if and only if $(x,y)\in \sim$, if and only if $x\sim y$. That $H$ is normal now follows because multiplication of cosets (equivalently, of equivalence classes modulo $\sim$) is well defined. QED
Definition. Let $A$ be an algebra (in the sense of universal algebra). A congruence on $A$ is an equivalence relation $\Phi$ on $A$ such that $\Phi$ is a subalgebra of $A\times A$.
THEOREM. Let $A$ and $B$ be algebras of the same type, and let $f\colon A\to B$ be a homomorphism; define $\Phi_f = \{(x,y)\in A\times A\mid f(x)=f(y)\}$. Then $\Phi_f$ is a congruence on $A$. Conversely, if $\Psi$ is a congruence on $A$, then there exists an algebra $\mathcal{B}$ and a homomorphism $\pi\colon A\to \mathcal{B}$ such that $\Psi=\Phi_{\pi}$.
Special case. Let me do the proof with semigroups. First, let $f\colon A\to B$ be a semigroup homomorphism; then $\Phi_f$ is an equivalence relation on $A$. To show it is a subsemigroup of $A\times A$, suppose $(x,y),(z,w)\in \Phi_f$. Then $f(xz) = f(x)f(z) = f(y)f(w) = f(yw)$, so $(xz,yw)\in\Phi_f$. Thus, $\Phi_f$ is a subsemigroup of $A\times A$, hence a congruence.
Conversely, suppose that $\Psi$ is a congruence. Let $\mathcal{B}=A/\Psi$, the set of equivalence classes of $\Psi$ in $A$. Let $[x]$ denote the class of $x$. We define a multiplication on $\mathcal{B}$ by $[a][b]=[ab]$. We must prove that this is well defined: suppose $[x]=[y]$ and $[z]=[w]$. Then $(x,y),(z,w)\in\Psi$, so $(xz,yw)\in\Psi$ (since $\Psi$ is a congruence). Therefore, $[xz]=[yw]$, so multiplication is well-defined. It is now easy to verify that multiplication is associative: $$[x]([y][z]) = [x][yz] = [x(yz)]=[(xy)z] = [xy][z] = ([x][y])[z].$$ Thus, $\mathcal{B}$ is a semigroup. The map $\pi\colon A\to \mathcal{B}$ given by $\pi(a)=[a]$ is easily verified to be a homomorphism, and $\Phi_{\pi}=\Psi$.
The proof for an arbitrary algebra with operations $f_1,\ldots,f_k$ of arities $a_1,\ldots,a_k$ (or even an infinite number of operations, each with finite arities) is essentially the same. QED
One can then prove the Isomorphism Theorems for "congruences" in exactly the analogous manner as for groups and rings, for any class of algebras.
In Groups and Rings, we can "get away" with not studying congruences because if $\Phi$ is a congruence on $G$, then we can let $N = \{ g\in G\mid (g,e)\in\Phi\}$, and it turns out that $\Phi$ is the equivalence relation induced by $N$. Essentially, a congruence tells you that certain things are equal, but in groups (and rings), instead of saying "$x$ is equal to $y$", we can say "$xy^{-1}$ is the identity". The normal subgroup (resp. the ideal) is the set of elements equivalent to the identity. In more general classes, such as semigroups, we don't have the luxury of expressing any identity in the form $w=e$, so we must actually keep track of both sides of the equality.
This is all part of Universal Algebra. I cannot recommend George Bergman's book highly enough if you want a great introduction to the subject (with a Category Theory flavor).
I cannot comment yet, but I thought I would elaborate on what is meant by being well-defined.
In your definition of the multiplication in $G/H$, you take cosets $xH$ and $yH$ and multiply by setting their product equal to $(xy)H$. But this means that you have chosen $x$ and $y$, and there will in general be other choices of $x$ and $y$ that give the same cosets. So if you have $xH$ = $x'H$ and $yH$ = $y'H$, you want to be sure that you also have $(xy)H = (x'y')H$, and it can be shown that this holds for all elements of $G$ iff $H$ is normal in $G$, so this is the only case where this multiplication is well-defined.