I am reading Elements of Information Theory by Cover and Thomas (2006) and struggle with the definition of mutual information for continuous random variables (Chapter 9: Differential Entropy). For two random varibles with a joint pfd $f(x, y)$, they define the mutual information as \begin{equation} I(X;Y) = \int f(x, y) \log \frac{f(x, y)}{f(x)f(y)} \text{d}(x, y). \end{equation} Later, they give a more general definition using mutual information of discrete random variables as \begin{equation} I(X;Y) = \text{sup}_{P, Q} I([X]_P; [Y]_Q), \end{equation} where $[X]_P$, resp. $[Y]_Q$ is a quantization of $X$, resp. $Y$ w.r.t. to finite partition $P$ of $X$, resp. $Q$ of $Y$. Now they say, that the definitions are equivalet for random variables with density and that it can be shown similarly to the way they show that the mutual information of continuous random variables is the limit of mutual information of their quantized version. For that they use a theorem stating that $H(X^{\Delta}) + \log\Delta \rightarrow h(X)$ for $\Delta \rightarrow 0$, where $\Delta$ is the length of a bin used for uniform quantization of $X$ and $X^{\Delta}$ is the corresponding quantized version of $X$. But since the general definition uses any partition, not necessarily the uniform one and without the uniformity, we do not have the limit of the quantization, I really cannot figure out the proof of equivalency. Can anyone help?
2026-02-23 10:03:25.1771841005
Equvivalent definitions of mutual information of continuous random variables
40 Views Asked by Bumbble Comm https://math.techqa.club/user/bumbble-comm/detail AtRelated Questions in INFORMATION-THEORY
- KL divergence between two multivariate Bernoulli distribution
- convexity of mutual information-like function
- Maximizing a mutual information w.r.t. (i.i.d.) variation of the channel.
- Probability of a block error of the (N, K) Hamming code used for a binary symmetric channel.
- Kac Lemma for Ergodic Stationary Process
- Encryption with $|K| = |P| = |C| = 1$ is perfectly secure?
- How to maximise the difference between entropy and expected length of an Huffman code?
- Number of codes with max codeword length over an alphabet
- Aggregating information and bayesian information
- Compactness of the Gaussian random variable distribution as a statistical manifold?
Related Questions in MUTUAL-INFORMATION
- Is there a chain rule for Sibson's mutual information?
- Why does the common information $C\equiv X\wedge Y$ give $H(X)=H(CX)$?
- $X \to Y \to Z$ PGM with $X,Y,Z ~ \text{MVN}(0,\Sigma)$. What is mutual information $I(X;Z)$? (Cover & Thomas 8.9)
- Derivative of a mutual information for a Gaussian channel
- Intuition on Mutual information in XAI
- Need help finding the stationary points of a function
- Computing covariances of features/landmarks in environment in scan-matching algorithm
- Total Correlation is difference of relative entropies in general?
- What is the mutual information between ingredients of a mixture of Gaussians?
- Characterization of functions that have the data processing inequality
Related Questions in CONTINUOUS-VARIABLES
- True or false: For continuous random variable $X$, if $E(X)$ has a closed form, then $P(X<E(X))$ has a closed form.
- Continuous conditional probability on fixed given value
- Derivation of expected value of sample variance
- Is the Lambert W function the Newton flow of the exponential function?
- Conditioning on a Continuous Variable with a Discrete Value
- Should discrete and continuous time models give the same results?
- What is the analogue of $P(A) = \sum_\sigma P(A , \Sigma = \sigma )$ for when $\Sigma$ is a continuous random variable?
- The combined probability of a continuous variable that is also a parameter for a discrete random variable.
- Simple algebra of integral
- Finding value of m, where m is the bound of a pdf such that it gives the median value.
Trending Questions
- Induction on the number of equations
- How to convince a math teacher of this simple and obvious fact?
- Find $E[XY|Y+Z=1 ]$
- Refuting the Anti-Cantor Cranks
- What are imaginary numbers?
- Determine the adjoint of $\tilde Q(x)$ for $\tilde Q(x)u:=(Qu)(x)$ where $Q:U→L^2(Ω,ℝ^d$ is a Hilbert-Schmidt operator and $U$ is a Hilbert space
- Why does this innovative method of subtraction from a third grader always work?
- How do we know that the number $1$ is not equal to the number $-1$?
- What are the Implications of having VΩ as a model for a theory?
- Defining a Galois Field based on primitive element versus polynomial?
- Can't find the relationship between two columns of numbers. Please Help
- Is computer science a branch of mathematics?
- Is there a bijection of $\mathbb{R}^n$ with itself such that the forward map is connected but the inverse is not?
- Identification of a quadrilateral as a trapezoid, rectangle, or square
- Generator of inertia group in function field extension
Popular # Hahtags
second-order-logic
numerical-methods
puzzle
logic
probability
number-theory
winding-number
real-analysis
integration
calculus
complex-analysis
sequences-and-series
proof-writing
set-theory
functions
homotopy-theory
elementary-number-theory
ordinary-differential-equations
circles
derivatives
game-theory
definite-integrals
elementary-set-theory
limits
multivariable-calculus
geometry
algebraic-number-theory
proof-verification
partial-derivative
algebra-precalculus
Popular Questions
- What is the integral of 1/x?
- How many squares actually ARE in this picture? Is this a trick question with no right answer?
- Is a matrix multiplied with its transpose something special?
- What is the difference between independent and mutually exclusive events?
- Visually stunning math concepts which are easy to explain
- taylor series of $\ln(1+x)$?
- How to tell if a set of vectors spans a space?
- Calculus question taking derivative to find horizontal tangent line
- How to determine if a function is one-to-one?
- Determine if vectors are linearly independent
- What does it mean to have a determinant equal to zero?
- Is this Batman equation for real?
- How to find perpendicular vector to another vector?
- How to find mean and median from histogram
- How many sides does a circle have?