I have an operation $f$ which takes two numbers $A$ and $B$ and returns a symmetric difference of digits of these two numbers. For example having $453$ and $1134$ the operation will produce a set {1, 5}.
The problem I am failing to solve is the following. Having a set $X$ and a number $N$ I need to calculate how many pairs of numbers below $N$ will produce me the same set $X$ after applying to them my operation $f$.
For example having a set {1, 2, 6} and a number $235$ the following pairs would be valid: $(1, 26)$, $(16, 222)$ and many more (total $268$ based on my quick program check).
My only approach was to divide a set into various two subsets: for example {1, 2, 6} into ({1} and {2, 6}), ({2} and {1, 6}) and others. Then looking in how many ways can I add same elements to both parts. Needless to say that this left me nowhere.
I suspect the problem to be really messy due to the very detailed constraints involved.
On the number of subset pairs
The number of pairs of subsets of $X=\{x_1,x_2,...,x_n\}$ is $2^{n-1}$ since to every subset $A\subseteq X$ we can assign an $n$-bit string with a $0$ at position $i$ if $x_i$ is not in the subset and a $1$ if it is. If the $0$'s and $1$'s are swapped we get the bit-string corresponding to the complement $A^c$. Thus each pair $A,A^c$ corresponds to a pair of distinct $n$-bit strings and we have $2^n/2=2^{n-1}$ such bit-string pairs.
One simple way of identifying all subset pairs would be to consider all sets $A$ have a bit-string representation starting with a $0$ so that the counterpart $A^c$ has a representation starting with a $1$. Then we are essentially iterating through $n$-bit strings starting with $0$ followed by all possible $(n-1)$-bit strings in turn.
Regarding the bound on integer sizes
The problem of conforming to the bound $N$ is somewhat more messy. The problem would be more clean if $N$ was a power of $10$ so that $N$ simply specified the number of digits allowed. So let us consider the somewhat cleaner case of $N=10^k$ so that solution pairs consist of $k$-digit integers.
If we have a division of $X$ into two sets $A,A^c$ in which $s:=\max(|A|,|A^c|)$ is less than $k$, we can add at most $\min(k-s,10-n)$ extra digits not from $X$ when forming our solution pair. Say we add $m\leq\min(k-s,10-n)$ extra digits.
Now for the number in our solution contributing with the digits from $A$ we have $|A|+m$ mandatory digits that should occur at least once and in the remaining $k-(|A|+m)$ places we can choose repeating any of the $|A|+m$ digits. And then again it becomes messy.
I will leave it here for now, but to apply my above analysis to your example let $X=\{1,2,6\}$. We have the subset pairs $$ \begin{align} 000&\mapsto(A,A^c)=(\emptyset,\{1,2,6\})\\ 001&\mapsto(A,A^c)=(\{6\},\{1,2\})\\ 010&\mapsto(A,A^c)=(\{2\},\{1,6\})\\ 011&\mapsto(A,A^c)=(\{2,6\},\{1\}) \end{align} $$
Then if we consider the division $\{6\},\{1,2\}$ for instance and take $k=3$ we see that $s=2$ so that we may introduce at most $m\leq k-s=1$ extra digits. So from the first set $A$ we can form numbers $6,x6,6x,xx6,x6x,6xx,66,x66,6x6,66x,666$ with $x\in\{3,4,5,7,8,9,0\}$. And then it is even a little more messy beacuse of the digit $0$ which yields a couple of special cases.