How to prove a differential entropy is not scale invariant?

Question

For example, $S(X)=-E_X(\log(f_X))=-\int_{-\infty}^{+\infty}f_X(x)\log(f_X(x))dx$ A transformation of X changes the result:$S(aX)=S(X)+\log|a|$ and more in general $S(g(X))=S(X)+\int_{-\infty}^{+\infty}f_X(x)\log(|\frac{\partial g}{\partial x}|)dx$.

Answer 1

You have correctly shown that the differential entropy of a random variable is not scale invariant. I encourage you not to spend any more time trying to prove that it is invariant under this transformation.

If you want a scale invariant measure for the entropy of a continuous random variable you need to use a different related quantity, for example the relative entropy.

Here is a proof that the relative entropy for the random variables $P$ and $Q$ is invariant under coordinate transformations (and hence under scale transformations).

The definition for the relative entropy is $$D_{KL}(P||Q) := \int dx f_P(x) \log\left(\frac{f_P(x)}{f_Q(x)}\right).$$ Under a change variable from $x$ to $y$, the pdfs of $P$ and $Q$ change as $$f_P(y)dy = f_P(x)dx\qquad\qquad f_Q(y)dy=f_Q(x)dx.$$ (Note that in this notation, $f_P(y)$ is a different function from $f_P(x)$). Then changing variable from $x$ to $y$ in the integral, we see that $$D_{KL}(P||Q) = \int dx f_P(x) \log\left(\frac{f_P(x)}{f_Q(x)}\right) = \int dy f_P(y) \log\left(\frac{f_P(y)\left|\frac{dy}{dx}\right|}{f_Q(y)\left|\frac{dy}{dx}\right|}\right).$$Hence we arrive at$$D_{KL}(P||Q) =\int dy f_P(y) \log\left(\frac{f_P(y)}{f_Q(y)}\right),$$showing invariance under coordinate transformation as required.

Answer 2

I believe the task is to show the general condition that differential entropy is not scale invariant. That the differential entropy of a random variable X is not the same as the differential entropy some function of the same random variable X. To prove the final line of the question.