How to measure the independence of random variables?

Question

I need a function, which measures, how independent are two random variables.

Ideally,

• It should give 0, if the variables are completely independent ($\rm{P(AB)=P(A)P(B)}$).
• It should give 1, if the variables are completely dependent ($\rm{A=B}$).

Obviously also a reverse result is okay.

After digging the net and my old remembers, I found something like $\rm\frac{P(AB)}{P(A)P(B)}$. This is trivially correct for complete independency. But it gives $\rm\frac{1}{P(A)}$ if $\rm{A=B}$, what is not what I want.

Answer 1

The correlation coëfficient of random variables $\mathbf1_A$ and $\mathbf1_B$ is what comes to mind: $$\rho\left(\mathbf{1}_{A},\mathbf{1}_{B}\right)=\frac{\mathsf{Cov}\left(\mathbf{1}_{A},\mathbf{1}_{B}\right)}{\sqrt{\mathsf{Var}\left(\mathbf{1}_{A}\right)}\sqrt{\mathsf{Var}\left(\mathbf{1}_{B}\right)}}=\frac{P\left(A\cap B\right)-P\left(A\right)P\left(B\right)}{\sqrt{P\left(A\right)\left(1-P\left(A\right)\right)P\left(B\right)\left(1-P\left(B\right)\right)}}$$ Here $\mathbf1_A$ is a random variable that takes value $1$ if $\omega\in A$ (i.e. if event $A$ occurs) and takes value $0$ otherwise.

Answer 2

The correlation coefficient between two random variables $X$ and $Y$ is a measure of linear relationship between $X$ and $Y$. For example, $Y$ can be a nonlinear function of $X$ and still $\operatorname{Corr}(X,Y)=0$. A better way of measuring the dependence between $X$ and $Y$ would be normalized mutual information: $$ I^*(X,Y)=\frac{I(X,Y)}{\sqrt{H(X)H(Y)}} \in [0,1], $$ where $I(X,Y)=D_{KL}(P_{X,Y}||P_X\otimes P_Y)$ is the mutual information between $X$ and $Y$, and $H(X)$ and $H(Y)$ are the corresponding entropies. When $X$ and $Y$ are independent, $I(X,Y)=0$ and, therefore, $I^*(X,Y)=0$. When $X=Y$, $I^*(X,Y)=1$ as required.