Essentially, what does $G(a,b)$ mean?
I have seen it in the context of matrix representations some Lie Group $G(a+b)$ being decomposed into submatrices that each transform as representations of $G(a)$ and $G(b)$. Such a representation is then said to be an element of $G(a,b)$.
This is my heuristic definition, based purely on my observations of it being used in theoretical physics (specifically string theory) literature. It's not rigorously defined anywhere and not justification is given for what it means, I've had to just infer it. This bothers me as I feel quite strongly that if I am to use any mathematical notations or techniques then I should understand their strict mathematical definitions.
Example:
Consider a $d-1$ rank square symmetric matrix $A^{ij}$ and a vector $B^k$ with $d-1$ elements, with $i,j,k \in [1,2,...,d-1]$. One can then form a traceless, symmetric, matrix of rank $d$:
$$
M^{\mu \nu} =
\begin{pmatrix}
A^{ij} & B^{k} \\
B_k & -\mathrm{Tr}[A]
\end{pmatrix},~~~\textrm{ with } \mu,\nu \in [1,2,...,d]
$$
This matrix forms a symmetric, traceless, representation of the group $SO(d)$. Let us write:
$$
\begin{align}
M'^{\mu\nu} &= U^\mu_{~\alpha} M^{\alpha \beta} U_\beta^{~\nu} \\
&= U^\mu_{~i} A^{ij} U_j^{~\nu} + U^\mu_{~k} B^{k} U_d^{~\nu} + U^\mu_{~d} B^{k} U_k^{~\nu} - U^\mu_{~d} U_d^{~\nu} \mathrm{Tr}[A]
\end{align}
$$
With $U$ some real unitary matrix. Hence we see that the submatrices of $M$ each transform as:
$$
\begin{align}
M'^{i j} &= U^i_{~k} A^{kl} U_l^{~j} \\
M'^{k d} &= U^{d}_{~d} B^{l} U_l^{k} \\
M'^{d d} &= - (U^{d}_{~d})^2 \mathrm{Tr}[A]
\end{align}
$$
Where $U^d_{~d}$ is just a real number and so the submatrices transform as a matrix, vector, and scalar. I understand all of this, it's rather trivial once the matrix $M$ has been defined, but what I don't follow is that authors traditionally now write:
Thus $A \oplus B$ is in the subgroup $SO(1, d-1)$.
I've found this query one that's difficult to look up, since I don't know the nomenclature for referring to groups written this way.
There's no general sense of "groups $G(a,b)$", only a very short list of special cases: $O(p,q)$, $SO(p,q)$, $U(p,q)$, $SU(p,q)$, and a few others, in addition to the single-argument groups like $SL(n)$ and $GL(n)$.
The apparent two (or more) -argument groups are orthogonal and unitary groups of signature $p,q$, generalizing the (compact) orthogonal and unitary groups $O(n)$ and $U(n)$, which are probably more familiar.
By definition, $O(p,q)=\{g\in GL(p+q,\mathbb R): g^t\pmatrix{1_p & 0 \\ 0 & -1_q}g=\pmatrix{1_p & 0 \\ 0 & -1_q}\}$. The number of pluses and minuses, any order, is the signature. Yes, we can use matching block decompositions to express matrices in this group. In particular, the subgroup of $O(p,q)$ of elements that are block-diagonal $\pmatrix{a & 0 \\ 0 & d}$ exactly consists of those with $a\in O(p)$ and $d\in O(q)$.
In fact, it is Sylvester's Inertia Theorem that given any symmetric square invertible real matrix $Q$, by a change of coordinates $Q\to A^t Q A$ for some square, invertible $A$, we can make $A^tQA$ diagonal with entries $\pm 1$, and the number of $+1$'s is completely determined, as is the number of $-1$'s.