Why a matrix with a low entropy will be sparser?

Question

The entropy of a matrix $\mathbf{P}$: $$H(\mathbf{P})=-\sum_{i j} \mathbf{P}_{i j} \log \mathbf{P}_{i j}$$

Why a matrix with low entropy will be sparser, and with high entropy will be smoother?

Answer 1

Notice that, for the evaluation of the entropy:

$$0 \log(0) = 1 \log(1) = 0.$$

While in general

$$x \log(x) > 0,$$

for $x \in (0, 1)$.

Then:

• ${\bf P}_{ij} = 0$ stands for a zero entry of the matrix ${\bf P}$, then the higher is the number of ${\bf P}_{ij} = 0$, the sparser ${\bf P}$ will be;

• Since ${\bf P}$ is row-stochastic, ${\bf P}_{ij} = 1$ stands for row composed by one "1" entry, and all others are "0"; again, this is related to the sparsity of the matrix.