Distributions satisfying expectation condition

Question

Let $q$ be a continuous density on $\mathbb{R}^d$. I am interested in the set $\mathcal{S}$ of continuous distributions $p$ that have the same support as $q$ such that:

$E_{X\sim q}[\log p(X)] = E_{X\sim p}[\log p(X)]$

Clearly $q \in \mathcal{S}$ and if the support of $q$ is bounded, then the uniform distribution is also in $\mathcal{S}$ as its log-density is constant. Are there any other elements in $\mathcal{S}$? Is there a way to characterize every element in $\mathcal{S}$? Thank you!

Answer 1

This is only a partial answer which shows that for generic $q,$ $\mathcal{S}$ has non-trivial members.

---

Working with compactly supported densities, $\mathcal{S}$ is equivalent to the set of $p$ such that $$D(q\| u) = D(q \| p) + D(p \| u),$$ where $u$ is the uniform density on the support of $q$ and $D$ is KL-divergence. The cases of $p = q$ and $p = u$ are clear above.

The form reminds me of the pythagorean theorem for linear families[1]: Let $\mathcal{L}$ be a linear family, $q \in \mathcal{L},$ and $r$ an arbitrary distribution. Then the projection $r^*$ of $r$ onto $\mathcal{L}$ [2] satisfies $$ D(q \| r) = D(q\| r^*) + D(r^* \|r).$$

Obviously one wants to set $r = u$ above. In principle $q$ can lie in a number of linear families. Let us construct a few of them (I tacitly assume the support to be a subset of $\mathbb{R},$ but this can easily be extended):

1. Family generated by $f_1 = 1, \alpha_1 = 1$; clearly the projection is $u.$
2. By compact support, all moments exist and the MGF is smooth near 0 (so moments determine the distributions). The family generated by the intersection of the decreasing sequence of families generated by $(\{x^n\}_{n \le k},\{\mathbb{E}_q[X^n]\}_{n \le k})$ will yield the projection $q.$ (which will be the only member of the family).
3. An obvious third case is the family generated by $(x, \mathbb{E}_q[X]).$ For instance, for $q(x) = x \mathbf{1}\{x \in [0,1]\},$ the projection of $u$ onto this family should be (i haven't properly proved this, and I'm crap at calculus of variations anyway. Please check) $$p(x) \propto e^{bx} \mathbf{1}\{ x\in [0,1]\}$$ where $b$ is chosen so that $\mathbb{E}_p[X] = \mathbb{E}_q[X].$

---

We can generalise the above in the obvious way - suppose $f$ is a $q$-integrable function, such that $\exists c$ such that $$\int_{\sigma} f(x) \frac{e^{c f(x)}}{Z } = \mathbb{E}_q[f(X)],$$ where $\sigma$ is the support of $q$ and $Z = \int_{\sigma} e^{cf(x)}$ is a normalising constant. The set of densities $\{ e^{cf(x)}/Z \} \subset \mathcal{S}.$

An interesting question (towards a characterisation of $\mathcal{S}$) is if every $p$ in $\mathcal{S}$ is a projection of $u$ onto a linear family containing $q$. This may involve using the continuity of $q$, which (I may be incorrect here) we haven't really used above (except to assert non-triviality of some projections).

---

A generic treatment of I-projections (like [2]) and linear and exponential families (and much more) can be found in this excellent monograph - https://users.renyi.hu/~csiszar/Publications/Information_Theory_and_Statistics:_A_Tutorial.pdf . Consult chapter 3.

---

[1] Given a set of functions $\{f_i\}_{i = 1:k}$ and corressponding constants $\alpha_i,$ the linear family generated by $(\{f\},\{\alpha\})$ is the set of distributions $\{p: \mathbb{E}_p[f_i(X)] = \alpha_i\}$ for each $i$.

[2] $r^* = \arg \min _{p \in \mathcal{L} } D(p\|r)$