For any sets $X,Y\neq \emptyset$ if we define a permutation group $G(X,Y)$ on the set $X\cup Y$ as follows: $$G(X,Y)=\left\{\sigma\in \text{Sym}(X\cup Y):\forall x,y\left[(x,y)\in X\times Y\iff (\sigma(x),\sigma(y))\in X\times Y\right]\right\}$$
Then I'm pretty sure $G(X,Y)$ can be expressed in terms of symmetric groups on subsets of $X\cup Y$ using various group theoretic products between them. What I first did was define the three sets $A,B,C$ by letting $A=X\setminus Y$ and $B=Y\setminus X$ with $C=X\cap Y$ so all these sets were pairwise disjoint, with $A\cup B\cup C=X\cup Y$ and we could express the cartesian product of $X$ with $Y$ as:
$$X\times Y=(A\cup C)\times (B\cup C)=(C\times C)\cup(C\times B)\cup(A\times C)\cup(A\times B)\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }\text{ }$$$$=\begin{cases}(A\times B)&\text{ if }(A\neq \emptyset)\land (B\neq \emptyset)\land (C=\emptyset)\\(C\times C)&\text{ if }(A=\emptyset)\land(B=\emptyset)\land (C\neq \emptyset)\\(A\cup C)\times (A\cup C)&\text{ if }(A\neq \emptyset)\land (B=\emptyset)\land (C\neq\emptyset)\\(B\cup C)\times (B\cup C)&\text{ if }(A=\emptyset)\land(B\neq\emptyset)\land (C\neq \emptyset)\\(C\times C)\cup(A\times C)&\text{ if }(A\neq\emptyset)\land(B=\emptyset)\land (C\neq \emptyset)\\(C\times C)\cup(C\times B)&\text{ if }(A=\emptyset)\land(B\neq\emptyset)\land (C\neq \emptyset)\\(C\times C)\cup(C\times B)\cup(A\times C)\cup(A\times B)&\text{ if }(A\neq\emptyset)\land(B\neq\emptyset)\land (C\neq \emptyset)\end{cases}$$
Now noting that the permutations in $G(X,Y)$ fix the sets $A,B$ and $C$ with:
$$\forall (v,\sigma)\in (X\cup Y)\times G(X,Y)\left[\sigma(v)\in A\iff v\in A\right]$$ $$\forall (v,\sigma)\in (X\cup Y)\times G(X,Y)\left[\sigma(v)\in B\iff v\in B\right]$$ $$\forall (v,\sigma)\in (X\cup Y)\times G(X,Y)\left[\sigma(v)\in C\iff v\in C\right]$$
I believe we can express $G(X,Y)$ in each of these cases as follows:
1. $G(X,Y)\cong \text{Sym}(X\setminus Y)\text{Sym}(X\setminus Y)$
2. $G(X,Y)\cong \text{Sym}(X\cap Y)$
3. $G(X,Y)\cong \text{Sym}(X)$
4. $G(X,Y)\cong \text{Sym}(Y)$
5. $G(X,Y)\cong \text{Sym}(X\cap Y)\text{Sym}(X\cap Y)$
6. $G(X,Y)\cong \text{Sym}(X\cap Y)\text{Sym}(Y\cap X)$
7. $G(X,Y)\cong \text{Sym}(X\setminus Y)\text{Sym}(Y\setminus X)\text{Sym}(X\cap Y)$
Since if we consider the digraph $D=(X\cup Y,X\times Y)$ then our group is just $\text{Aut}(D)=G(X,Y)$ where $\text{Aut}(D)$ is the automorphism group of our digraph $D$ now with this set of source vertices in $D$ is simply the set $A$ while the set of sink vertices in $D$ is $B$ and lastly the set of internal vertices in $D$ is $C$ further because every digraph automorphism must map sink vertices to sink vertices, source vertices to source vertices and internal vertices to internal vertices I deduced that each permutation fixed each of the three sets $A,B$ and $C$. From there the result seems kinda obvious, however I'm not totally sure because I haven't done stuff like this in a while.
Update: Not thinking haven't slept in a while.... alright so clearly I didn't need to separate it out into eight cases not sure why I did that. Anyway under convention taking $\text{Sym}(\emptyset)$ to be a trivial group up to isomorphism, we can write $G(X,Y)$ out explicitly as:
$$G(X,Y)\cong \text{Sym}(X\setminus Y)\times \text{Sym}(Y\setminus X)\times\text{Sym}(X\cap Y)$$
Because by the previous reasoning $G(X,Y)$ induces a natural action on the block system $\{A,B,C\}$