Suppose $L$ is a language over $\{ (, ), [, ]\}$ defined by the formal grammar
$$S \to \Lambda$$ $$S \to (S)S$$ $$S \to [S]S$$
Suppose $L(n, m)$ is the sublanguage of all words containing exactly $n$ letters ‘$($’ and $m$ letters ‘$[$’. Let’s define $C(n, m) := |L(n, m)|$.
Is there some sort of explicit formula for $C(n, m)$?
I know only three facts about that sequence:
$$C(n, m) = C(m, n)$$
$C(n, 0)$ is the $n$-th Catalan’s number.
The generating function of the sequence $C(n, m)$ which is $G(x, y) := \sum_{n=1}^\infty \sum_{m=1}^\infty C(n, m)x^ny^m$ satisfies the equality
$$G(x, y) = xyG(x, y)^2 + f(x) + f(y) + 1$$
where $f$ is the generating function for Catalan numbers.
However, I failed to derive anything else from that.
Every sentence in $L(n, m)$ can be uniquely generated from a sentence in $Dyck(n+m)$ by converting some selection of $m$ of the $n+m$ open parentheses and their matching close parentheses into brackets. So $C(n,m)$ is just $\binom{n+m}{m}C_{n+m}$.
Since the grammar of $L$ is deterministic, a similar argument can be made directly from the grammar. A derivation of a sentence in $L(n, m)$ contains exactly $n$ applications of $S\to(S)S$ and $m$ applications of $S\to[S]S$. This derivation can be produced from a derivation of a sentence in $Dyck(n+m)$ by replacing $m$ of the $n+m$ applications of $S\to(S)S$ with $S\to[S]S$. This is precisely the same argument but the proof may be simpler.