In what situations does Shannon entropy increase or decrease?

Question

Typically when we measure information/Shannon entropy (differential entropy), it is a snapshot value at a specific point in time, so there is no talk about whether it will grow or decrease. A random variable just possesses that amount of entropy like the number of ears on a dog.

If we reconcile Shannon entropy with time entropy, which instead dynamically grows larger as $t\rightarrow \infty$, from the perspective that we as humans only experience time going forwards not backwards,

in what situations would we find informational disorder/transmission (information entropy) increase or decrease, given that Shannon entropy is commonly viewed as a non-dynamic snapshot value as if it were just a random variable's intrinsic property that doesn't change?

Answer 1

Like you said, entropy is a property of a random variable's distribution. So, if you want a time-variant entropy, you need to have some time-variance in your distribution, too.

One way to look at this is repeated experiments. For example, the entropy of a coin flip doesn't change over time, but you can look at the entropy of a sequence of coin-flips, which is a growing binary sequence, that changes over time. The entropy of this sequence changes over time as the sequence grows. You actually end up with very interesting phenomena in this case, a concentration of measure around what is called a typical set that contains equally likely outcomes (see asymptotic equipartition property if you're curious) and the probability of all outcomes outside this set vanishes.

Another way to have a time-variant entropy is having a distribution that evolves with time. You can define a family of random variables that is parametrized by the time index. As a simple example, let $X_t \sim \text{Uniform}(\{1/k\}_{k=1}^t)$. So, the set of values $X_t$ can take grows over time and hence entropy keeps increasing.

However, note that, the entropy doesn't always have to increase over time. You can conversely define a family, where the uncertainty decreases over time. For example, $X_t \sim \text{Uniform}([0,1/t])$ gives you $H(X_t) = \log (1/t)$.

Answer 2

Assume you are given a gradient flow \begin{equation} \dot{x}=-\nabla V(x). \end{equation} Then, $V$ decreases along solutions since \begin{equation} \frac{d}{dt}V(x)=\nabla V(x)\cdot \dot{x}=-|\nabla V(x)|^{2}<0. \end{equation} A prototypical example of an evolution of probability measures for which you can observe a decrease of entropy is the heat flow \begin{equation} \partial_{t}\mu=\Delta\mu. \end{equation} To see why, you can either directly differantiate in $t$, $S(\mu_{t})$ with $S=\int\mu\log\mu$ or you deduce it from the fact that the heat equation can be seen as gradient flow of the entropy in Otto's Wasserstein calculus, i.e. formally \begin{equation} \dot\mu=-\nabla^{W}S(\mu), \end{equation} with $\nabla^{W}$ being the Wasserstein gradient.