KL divergence minimization

76 Views Asked by Bumbble Comm At 26 Mar 2026 - 5:51

I have seen different derivations of KL divergence problems. Up to here, I understand everything:

$-\sum \mathbb Q(z) * log (\frac{P(x,z)}{Q(z)})+\sum \mathbb Q(z)*log(P(x))$

Then in next step we get:

$-\sum \mathbb Q(z) * log (\frac{P(x,z)}{Q(z)})+1*log(P(x))$

See:

$\sum \mathbb Q(z) = 1$

But this just can not be true:

$\\x1,x2,x3 = .1, .5, .4 \\ \\log_{2}(x1,x2,x3) = -3.2, -1, -1.32 \\ \\-\sum log_{2}(x1,x2,x3) = -5,64 \\ \\z1,z2,z3 = .2, .3, .5 \\ \\Q(z) * log(P(x)) = -1,62 \\ \\Q(z) * log(P(x)) != -\sum log_{2}(x1,x2,x3)$

I am sure I am missing something, but I just can not understand it what.

additional information:

I do not understand How Q(z) * log(P(x)) becomes log(P(x)). My question would be why Q(z) * log(P(z,x)/Q(z)) doesnt become just log(P(z,x)/Q(z)) if sum of Q(z) is always one as we know.

In lower part I have presented this with two sets of random variables for Q and P distribution. When applying Q(z) * log(P(x)) with these random variables, I get completely different result if there would be only log(P(x)).

qx = x only if q = 1, but in my case as it is evident qx is not x, therefore q is not one.

Original Q&A

KL divergence minimization

Related Questions in BAYESIAN

Trending Questions

Popular # Hahtags

Popular Questions