I'm working on an unassessed course problem,
Use the Orbit-Stabiliser Theorem to show that the set of matrices $$ M = \begin{pmatrix} a&b \\ c&d \end{pmatrix} \in \operatorname{GL}_2(\mathbb{R}) $$ with $a+c=b+d=1$ form a subgroup of $\operatorname{GL}_2(\mathbb{R})$.
I've written,
Let $$G=\operatorname{GL}_2(\mathbb{R}), \quad B=\begin{pmatrix}e&f\\ g&h\end{pmatrix}\in G, \quad A=\begin{pmatrix}a&b\\ c&d\end{pmatrix}\in M;$$ clearly $G$ acts on $M$ by matrix multiplication and $$ BA = \begin{pmatrix}e&f\\ g&h\end{pmatrix} \begin{pmatrix}a&b\\ c&d\end{pmatrix} = \begin{pmatrix}ea+fc& eb+fd \\ ga+hc&ga+hd\end{pmatrix}. $$ Now, clearly $\operatorname{orb}_G(A)=G$, and \begin{align} & \begin{aligned}[t] & (ea+fc)+(ga+hc) = (e+g)a+(f+h)c, \\ & (eb+fd)+(ga+hd) = (e+g)b+(f+h)d \end{aligned} \\[0.5em] \therefore \; & \begin{aligned}[t] & \operatorname{stab}_G(A) = \{B\in G \text{ s.t. } e+g=f+h=1\}=M, \\[0.5em] \end{aligned} \end{align} so, by the Orbit-Stabiliser Theorem, we have $$\infty = |\operatorname{orb}_G(A)| = \frac{|G|}{|\operatorname{stab}_G(A)|} = \frac{\infty}{\infty} \text{ or something.}$$
Is there a way to finish this argument validly?
I don't see a way to use the orbit-stabilizer theorem, which is about a bijection between an orbit and the coset of a stabilizer of a point in that orbit, to prove that something is a (sub)group. However, perhaps the real intent here was to use that stabilizers of a group action are subgroups. With this, the proof is easy:
The group $\operatorname{GL}_2(\mathbb{R})$ acts on $\mathbb{R}^2$ from the right via $$ (x\quad y) \cdot \begin{pmatrix} a&b \\ c&d \end{pmatrix} = (ax+cy\quad bx+dy). $$ Now the stabilizer of the vector $(1\quad 1)$ with respect to this action is the set of matrices you are considering.