Do the set of all factorisable distributions form an exponential family?

161 Views Asked by At

A (natural) exponential family of probability distributions is defined as $$ p(x|\theta) = \exp\big(g(x) + \theta\cdot f(x) - \psi(\theta)\big),\tag{1} $$ where $\theta$ is a vector of parameters and $\psi = \log Z$ is chosen such that $p(x|\theta)$ is normalised.

Now consider two jointly distributed random variables $A$ and $B$ with finite support, and consider the set of all joint probability distributions such that $$ p(a,b) = p(a)p(b)\tag{2} $$ for all $a$ and $b$ in the support of $A$ and $B$. That is, the set of all distributions that factor into independent distributions for $A$ and $B$.

I suspect that the set of distributions defined by (2) is an exponential family. If this is the case, how can it be shown, and in particular, how can this family of distributions be explicitly expressed in the form (1)?

For bonus points, I suspect the same is also true of conditional independence relationships, e.g. $p(a,b|c) = p(a|c)p(b|c)$, so the bonus question is how to express this in the form (1), assuming it can be done.

1

There are 1 best solutions below

0
On

As often happens, the answer to this became clear shortly after posting the question.

It's easier to answer a slightly more general question. Suppose we have two exponential family distributions (over any support), one given by $p(a|\theta_A) = \exp\big(g_A(a) + \theta_A\cdot f_A(a) - \psi_A(\theta)\big)$ and the other by $p(b|\theta_B) = \exp\big(g_B(b) + \theta_B\cdot f_B(a) - \psi_B(\theta_B)\big)$. We now wish to show that their product, $$ p(a,b|\theta_A\theta_B) = p(a|\theta_A)p(b|\theta_B) $$ is also an exponential family. This is straightforward, since $$ p(a|\theta_A)p(b|\theta_B) = \exp\big(g_A(a)+g_B(b) + \theta_A\cdot f_A(a) + \theta_B\cdot f_B(b) - \psi_A(\theta_A) - \psi_B(\theta_B)\big), $$ so we can just write $$g(a,b) = g_A(a)+g_B(b),$$ $$f(a,b) = f_A(a)+f_B(b),$$ $$\psi(\theta_A,\theta_B) = \psi_A(\theta_A)+\psi_B(\theta_B)$$ and we have an exponential family with parameters given by the elements of both $\theta_A$ and $\theta_B$.

Since all discrete distributions with finite support can be expressed as exponential distributions, this is sufficient to show that the product of two discrete distributions with finite support is in fact an exponential distribution. This also makes it easy to write the product distribution explicitly in the form of an exponential distribution -- you simply write the two discrete distributions as exponential distributions in whichever way you prefer (there are more than one way to do it), and then form the product as above. If I remember later I will update this answer with an example, along with the answer to my bonus question.