Such binary operator returns an output in the same set as the two inputs and is a general computer program.
Such general computer program is used in the reduce phase of some computations I run. Knowing ahead of time whether my general computer program is commutative is a necessary requirement to decide whether the reduce data has be processed in a specific order or not.
I'd like to know whether it is possible to ascertain logically that such an operator is commutative without doing an exhaustive search of all the pair-wise combinations.
I assume you mean binary functions. If your range of inputs is finite, then you can check all possible input pairs to check commutativity. Otherwise, no; in general there is not an easy check for commutativity.