The formal definition of big-$O$ in two variables is $f(m,n) \in O(g(m,n))$ if and only if $f(m,n) \leq C g(m,n)$ for some $C>0$ and all $m,n$ larger than some threshold $N$.
In the analysis of algorithms, however, we often have some interdependence between $m$ and $n$ manifesting in some inequality. For instance, if we're considering algorithms on a connected undirected graph where $n$ is the number of vertices and $m$ the number of edges, we have the constraint $n-1 \leq m \leq {n \choose 2}$. Hence if, say, we establish that an algorithm on a connected undirected graph always takes at most $n^2 + m$ steps, we can say that the runtime of said algorithm is $O(n^2)$ since $m \leq n^2$.
However, obviously it is not the case that $n^2 + m \in O(n^2)$ if we go strictly by the definition in the first paragraph without any interdependence between $m$ and $n$. For instance, $n^2 + m$ dominates $n^2$ if we take $m=2^n$. Specifically, what is happening here is that in the above example, the domain of the runtime is being restricted to a certain constraint-set, namely $n-1 \leq m \leq {n \choose 2}$, and the $O$-relationship holds on this set but not on all of $\mathbb{N}^2$.
I have not actually seen a definition proposed in an algorithms textbook where such "constraint-sets" are considered. Here is a proposed definition.
We say a set $S \subset \mathbb{N}^{2}$ is absolutely unbounded if for every positive integer $N$, $S \cap \{(m,n) \in \mathbb{N}^2 : m,n>N\} \neq \varnothing$. Let $T(m,n)$ be the runtime of an algorithm with inputs of size $m,n$ defined on some absolutely unbounded constraint-set $S \subset \mathbb{N}^2$. In the example given above, we would have $S = \{(m,n) \in \mathbb{N}^2 : n-1 \leq m \leq {n \choose 2}\}$.
We say $T(m,n) \in O(g(m,n))$ if there is a $C>0$ and an $N \in \mathbb{N}$ such that for all $(m,n) \in S \cap \{(m,n) \in \mathbb{N}^2 : m,n>N\}$ we have that $T(m,n) \leq Cg(m,n)$.
Is this satisfactory? Are there better definitions? Better yet, are there references for definitions? I mean, I personally sort of use this definition in practice (e.g., coursework and research) but unless it's backed by numerous professionals in the field and peer-reviewed work there is naturally some concern that the definition is insufficient or flawed.