I previously asked a question about the factor lemma in Jech, and from the comments I realised that there is something earlier that I need to understand.
Let $Q$ be some forcing notion in $M$ and let $H$ be $Q$ generic over $M$ such that $H\in M[G]$ where $G$ is $Coll(\omega, <\lambda+1)$ and $|Q|<\lambda$.
I want to find some forcing notion $P$ and $K\subseteq P$ that is $P$ generic over $M[H]$ such that $M[H][K]=M[G]$.
From what I gathered my $P$ is the quotient forcing, but as far as I (thought) I knew, to define the quotient forcing I need some dense homomorphism $\pi: Coll(\omega, <\lambda+1)\to Q$.
What exactly is the definition of $P$ then?
Here is how the quotient forcing works. You can read up on all of this for example in Kunen's book (the 2011 version), pages 351-353.
You have forcing posets $Q$ and $P$ (note that I'm changing the meaning of $P$ here compared to your question) and $Q$ is a complete subposet of $P$ (not necessarily dense!). This means that every maximal antichain in $Q$ is still a maximal antichain in $P$.
Now if $G$ is $P$-generic over $M$, then $H := G \cap Q$ is $Q$-generic over $M$ and $M[H] \subseteq M[G]$. Now how to get from $M[H]$ to $M[G]$? This is the quotient forcing. Namely, in $M[H]$ you can define $$P/H := \{ p \in P : \forall q \in H (p \not\perp q) \},$$ in other words $P/H$ consists of the conditions which are compatible with every single member of $H$. The order on $P/H$ is the same as on $P$. It turns out that $G$ is $P/H$-generic over $M[H]$ and obviously $M[H][G] = M[G]$.
So far so good. But this was a very specific situation where we know that $Q$ is a complete subforcing of $P$. Now here comes the thing you might be actually asking if I understand correctly: We just have some arbitrary set $X \in M[G]$ (in your question now this is $H$, and in the original question this was called $X$) and we want to know how to get from $M[X]$ to $M[G]$.
Now here is where complete Boolean algebras are much more useful to describe what is happening. So let $P$ be a complete Boolean algebra (this is not a real restricition at all). For simplicity, first assume that $X$ is just a set of ordinals (this is actually all that is assumed in Jech's book in your original question). Then define $Q \subseteq P$ to be the complete Boolean subalgebra of $P$ generated by $$\{ [ \alpha \in \dot X ] : \alpha \in \operatorname{Ord}\}.$$ Here recall that $[\alpha \in \dot X]$ is the maximal condition that forces that $\alpha \in \dot X$ (these brackets usually have two lines in the books). So $Q$ is generated by the conditions that precisely decide what $X$ will be. Then, as $Q$ is (!) a complete subalgebra of $P$, and if we let $H := G \cap Q$, $X \in M[H]$, since we can decide a name for $X$. It turns out that $M[H] = M[X]$. And now we know how to get to $M[G]$ from $M[X] = M[H]$ by using the quotient forcing from before.
When $X$ isn't a set of ordinals there is no guarantee that this is possible, since $M[X]$ might not even be a model of ZFC (the axiom of choice could fail). But if it is a model of ZFC, then there is a set of ordinals $Y$ so that $M[X] = M[Y]$.