Can every factorization be represented by a Bayesian network?

Question

A Bayesian network is defined as a directed acyclic graph with a set of random variables as its nodes, and it satisfies two axioms,

1) Root nodes (nodes without parents) are independent.

2) Given a variable $X$ in the network, denote its parents (adjacent nodes with inbound edges to $X$) as $p(X)$. A RV $X$ is conditionally independent from all other RVs on $p(X)$. For example,

A Bayesian network represents a factorization of joint distributions. In the following, $\cal p(X)$ is the set of all parents of $X$ in the network.

My question is the converse of the above statement, i.e. can every factorization be represented by a Bayesian network? If not, can anyone help provide a counter example? Thank you!

Answer 1

One way to answer your question is, given such a factorization, construct a corresponding Bayesian network (i.e. one which represents such a factorization). This can always be done.

This is easiest of course if we have a finite graph, but the general principle should be applicable irrespective of the cardinality of $\mathcal{V}$.

1. For every element $X$ of the index set $\mathcal{V}$, draw/create a node.

2. For each $X$, draw an arrow/arc/directed edge from every member of $p(X)$ to $X$.

3. Assign the appropriate probabilities.

Remember that we can compute the (unconditional) probability distribution, given conditional probabilities and all of the joint probabilities (i.e. not just the joint distribution of all of the random variables together, but also the joint distributions of all subsets of random variables).

The issue is that we might not necessarily be able to infer uniquely the marginal joint probabilities given the total joint distribution and the conditional probabilities in this form, especially if we have two Markov equivalent Bayesian networks.

In other words, we might have a uniqueness issue here, with multiple possible marginal joint distributions implying the same total joint distribution and conditional distribution.

The given factorization only amounts to a specification of the total joint distribution and the conditional distribution, so any marginal joint distributions which generate the desired conditional distribution is an appropriate construction.

So, in short, we should expect that such a factorization can specify and thus be represented by a Bayesian network, but not uniquely. However, if we are given enough information about the marginal joint distributions, enough for example to resolve all ambiguities/multiple possibilities arising from Markov equivalence, then we could even construct a unique Bayesian network corresponding to the factorization.

Also keep in mind that the simplicity of this construction is due in part to the fact the simplicity of the Markov blankets implied by such a factorization. For other types of graphical models, for which the Markov blankets are more complicated, it might be more difficult to ensure that there exists an appropriate construction given any factorization.

Markov equivalence

http://www.multimedia-computing.de/mediawiki/images/5/55/SS08_BN-Lec2-BasicProbTheory_3.pdf

http://www.stats.ox.ac.uk/~steffen/teaching/gm09/dag.pdf

Markov blanket

https://en.wikipedia.org/wiki/Markov_blanket

https://en.wikipedia.org/wiki/Moral_graph