Mutual Information vs positive/negative correlations

419 Views Asked by At

As we can read on Wikipedia, "Unlike correlation coefficients, such as the product moment correlation coefficient, mutual information contains information about all dependence—linear and nonlinear—and not just linear dependence as the correlation coefficient measures."

Anyway, Pearson's correlation coefficient can assume both positive and negative values. E.g. a positive correlation means that if one variable increases the other also increases, while A negative correlation works the other way: if one increases the other decreases.

Is it possible to infer a similar result also by observing the Mutual Information or another function based on information entropy?

Thank you!

1

There are 1 best solutions below

0
On BEST ANSWER

No, the mutual information is always positive, and it's insensitive to "directions" in the sense above. For example, $I(X;Y)=I(X;-Y)$.

As another example, the same paragraph of the Wikipedia shows that the mutual information of two jointly gaussian variabless is $-\frac12 \log(1- \rho^2)$, hence it's insensitive to the sign of the correlation factor.