A binary operation is a calculation involving two elements of the set and returning another element of the set. Suppose it doesn't return an element of the set. What is the function called?
For example subtraction in the natural numbers. a - b with $a, b \in \mathbb{N}$ can return an element outside $\Bbb N$.
We don't have a specific name for when the returned value does not come from the same space as the inputs. To see why this idea doesn't deserve its own word, let's look at it more carefully.
If we want to let our reader know that the $-$ operation defined on naturals numbers doesn't return a natural number, we simply have to give them the function's signature, $-:\mathbb{N}\times\mathbb{N} \to \mathbb{Z}$.
In general we write $f:A \to B$ if $f$ takes inputs in $A$ and returns values in $B$. It is good practice to include the function signature in the definition of a function, especially when it is not immediately obvious what the $A,B$ should be. Because writers are typically very good about including the codomain $B$ explicitly when defining a function, there is not a good reason to give a name to types of functions $f: A \times A\to B$ where $B \neq A$.
In general, stick to clearly specifying the function signature whenever it is needed.