How to deconstruct Shannon Joint Entropy $H(X,Y,Z)$ equation for semi-related variables?

322 Views Asked by At

Background

The purpose of this is to produce a shuffle correction for Transfer Entropy estimation:

$$\operatorname{TE}_{x\to y}=H(X_{t+\tau})-H(X_\tau)-H(X_{t+\tau},Y_t,X_t)+H(X_t,Y_t)$$

In order to produce a shuffling correction, one would ideally calculate a shuffle correction such that only for $H(X_{t+\tau},Y_t,X_t)$ there would be a shuffled dataset, where the original relationships would only be destroyed between $X_{t+\tau}$ and $Y_t$, but not between $X_{t+\tau}$ and $X_t$ or $Y_t$ and $X_t$.

The Question

For clarity let's assume we wish to calculate Shannon Joint Entropy for $H(X,Y,Z)$, where the dataset is in the form:
X Y Z
1 0 0
1  1  1
0 0 1
1  0 1
0 0 0
0 0  1
1  1  0

It is possible to calculate $H(X,Y,Z)$ such that the original relationship between $X$ and the other two is destroyed, while the original relationship between $Y$ and $Z$ is maintained. This could be done simply by shuffling the order of values in $X$ column and then calculating the joint entropy as normal. Also, since $X$ is now independent from $Y$ and $Z$, the joint entropy is equal to the sum of independent entropies $H(X,Y,Z) = H(X) + H(Y,Z)$.

Is there a way to deconstruct the calculation of joint entropy $H(X,Y,Z)$ such that it would allow calculating joint entropy for $X$, $Y$ and $Z$ when $X$ and $Y$ are independent (original relationship destroyed), while $Y$ and $Z$ are dependent and so are $X$ and $Z$ (original relationship maintained)?