I am calculating p-value of mutual information (MI) between measurements of two discrete variables A and B by permutation test. The null hypothesis is that A and B are independent, as defined by MI.
The permutation test I implemented randomly permute the measurements of A, and calculate the MI between the permuted measurements of A and the original measurement of B. After doing this for several times, I get a distribution of MI. From this distribution and the actual MI between the non-permuted measurements, I get the p-value.
The problem is that I get an actual MI that is far lower than nearly all of the distribution of MI for some pairs of real world data.
But, the MIs from permutation tests are mostly MI between independent distributions. This is because the permutation is supposed to get rid of relationships between A and B. I thought the actual MI should be higher or equal to those MIs from permutation tests. This because there can be some relationship between A and B, which will lead to a higher actual MI than most of the MI from the permutation test. Or, A and B has no relationship, which will lead to an actual MI that is close to most of the MI from permutation test.
Is it normal to have an actual MI lower than most of the MI from permutation test? What is the meaning of this?