Intuitively, there seems to be a link between the (kind of) homomorphism between two algebraic structures and the mutual information between two variables. However, since I'm not a mathematician, it's hard to see or formalize the exact connection. Anyone who can help me with this?
For example, if we'd take each variable to represent a set and its order relation, could we say that the very existence of a homomorphic map between the two would make their $ MI \gt 0 $ ?
And the other way around: would an $ MI = 0 $ between those variables make the map between them a random one?
Let's look at two examples.
In the case of vector spaces we have linear maps, or vector space homomorphisms. These preserve scalar products (1) and are linear(2).
Let $f: V \to W$ be a linear map.
$$f\big(\lambda x \in V \big) = \lambda f(x) \in W\quad (1)$$ $$f\big(x+y \in V\big) = f(x) + f(y) \in W \quad (2)$$
So we see that the operation of addition and scalar multiplication is preserved between $V$ and $W$.
In the case of groups, we have group homomorphisms. Similarly these preserve group operations.
Let $g: A \to B$ be a group homomorphism and $*$ the group operation.
$$g\big(a_1 * a_2 \in A\big) = g(a_1) * g(a_2) \in B.$$
So, in a sense, we can say the "mutual information" of the operations is preserved when a homomorphism maps an element from one object to another.