I've been conducting a small experiment to test a few of my interpretations about mutual information, and I'm running into some difficulties.
I've created some MATLAB code that basically makes two sets of $10,000$ random, normally-distributed values, centred around a mean of $0$. I called these sets $X$ and $Y$. I then "bin" each of these sets into two histograms containing ten bins. From here, I can calculate the entropy of each set, and thus their "self" information; $$H(X) = I(X;X)$$ $$H(Y) = I(Y;Y)$$ Now, the next thing I wanted to test was the information between $X$ and $Y$. To do this, I created a bivariate histogram, which gives me a $10 \times 10$ grid of joint counts, which I can then divide by $10,000$ to get the joint probabilities of $X$ and $Y$ occurring at the same time. I then used the method shown here; http://www.scholarpedia.org/article/Mutual_information to calculate the information value.
Now, my intuition was telling me that, because $X$ and $Y$ are roughly the same, then $I(X;Y) = I(Y;X) \approx H(X) \approx H(Y)$. Now, it's definitely true that $I(X;Y) = I(Y;X)$, and I have shown that $H(X) \approx H(Y)$, but the values I get for $I(X;Y)$ are around $0.005$, whereas my entropy values are generally around $2.4$.
I'm not 100% sure where this discrepancy is coming from. Any insight into why this might be happening would be very appreciated!!
That's wrong. If $X$ and $Y$ are independent (as it seems here), then $$I(X;Y)=H(X)-H(X|Y) = H(X)-H(X)=0$$ (which is consistent with the intuitive notion of mutual information: if $Y$ is independent from $X$, then $Y$ gives me no information about $X$ - and viceversa). The value you've got ($0.005$) is very small relative to the marginal entropies - hence, it looks right.
BTW: the mutual information is symmetric, so $I(X;Y)=I(Y;X)$ always, it does not matter that they are identically distributed.