Probability of a gaussian distribution in another gaussian distribution

Question

Assume we have a Gaussian distribution $p(x) \sim \mathcal{N}(\mu_p,\Sigma_p)$

For any point $X$, it is easy to compute the density of $x$ in $p$: $$p(x) = \frac{1}{|2\pi \Sigma_p|^\frac{1}{2}}e^{-\frac{1}{2}(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)}$$

Now suppose that we have another Gaussian distribution $q(x) \sim \mathcal{N}(\mu_q,\Sigma_q)$

** What is the expectation of q(x) in p(x) . I mean, if we randomly sample a point $x$ from P, what is the **expected probability of x in Q?

NOTE: I need the solution in closed form.

Thanks!

Answer 1

Note that in this case we will have:

$$E_{x\sim p(x)}[q(x)]=E_{x\sim q(x)}[p(x)]=\int_{x\in \mathcal{X}} p(x)q(x)dx$$ $$=\int_{x\in \mathcal{X}} \left(\frac{1}{|2\pi \Sigma_p|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)}\right)\left(\frac{1}{|2\pi \Sigma_q|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)}\right)dx$$

$$=\int_{x\in \mathcal{X}} \frac{1}{|2\pi \Sigma_p|^\frac{-1}{2}|2\pi \Sigma_q|^\frac{-1}{2}}e^{-\frac{1}{2}\left((x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)+(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)\right)}dx$$

Now go ahead and, expand $(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)+(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)$ and regroup the the terms to get: $$(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)+(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)=x^TAx+b^Tx+d$$

Then work out the algebra to get $\Sigma$ and $\mu$ such that:

$$x^TAx+b^Tx+d=(x-\mu)^T\Sigma^{-1}(x-\mu)+F$$

After this you will have some thing like this:

$$E_{x\sim q(x)}[p(x)]=\frac{e^{\frac{-1}{2}F} |2\pi \Sigma|^\frac{-1}{2}}{|2\pi \Sigma_p|^\frac{-1}{2}|2\pi \Sigma_q|^\frac{-1}{2}} \int_{x\in \mathcal{X}}\frac{1}{|2\pi \Sigma|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)}dx$$

$\int_{x\in \mathcal{X}}\frac{1}{|2\pi \Sigma|^\frac{-1}{2}}e^{-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)}dx=1$, so you'll have:

$$E_{x\sim q(x)}[p(x)]=\frac{e^{\frac{-1}{2}F} |2\pi \Sigma|^\frac{-1}{2}}{|2\pi \Sigma_p|^\frac{-1}{2}|2\pi \Sigma_q|^\frac{-1}{2}} $$

Edit

$$(x-\mu_p)^T\Sigma_p^{-1}(x-\mu_p)+(x-\mu_q)^T\Sigma_q^{-1}(x-\mu_q)=x^T(\Sigma_p^{-1}+\Sigma_q^{-1})x+(-2\mu_p^T\Sigma_p^{-1}-2\mu_q^T\Sigma_q^{-1})x+(\mu_p^T\Sigma_p^{-1}\mu_p+\mu_q^T\Sigma_q^{-1}\mu_q)$$

Let $A=\Sigma_p^{-1}+\Sigma_q^{-1}$,$b=(-2\mu_p^T\Sigma_p^{-1}-2\mu_q^T\Sigma_q^{-1})^T$, and $d=\mu_p^T\Sigma_p^{-1}\mu_p+\mu_q^T\Sigma_q^{-1}\mu_q$.

Now we should find $\Sigma$, $\mu$, and $F$, we have :

$$(x-\mu)^T\Sigma^{-1}(x-\mu)=x^T\Sigma x-2\mu^T\Sigma^{-1}x+\mu^T \Sigma^{-1} \mu $$

Let: $$x^TAx+b^Tx+d=x^T\Sigma^{-1} x-2\mu^T\Sigma^{-1}x+\mu^T \Sigma^{-1} \mu +F$$

$$\Sigma^{-1}=A=\Sigma_p^{-1}+\Sigma_q^{-1}\Rightarrow \Sigma=\left(\Sigma_p^{-1}+\Sigma_q^{-1}\right)^{-1}$$

$$b^{T}=-2\mu^T\Sigma^{-1}\Rightarrow \mu=-\frac{1}{2}\Sigma b=-\frac{1}{2}\left(\Sigma_p^{-1}+\Sigma_q^{-1}\right)^{-1} (-2\mu_p^T\Sigma_p^{-1}-2\mu_q^T\Sigma_q^{-1})^T$$

$$\Rightarrow \mu=\left(\Sigma_p^{-1}+\Sigma_q^{-1}\right)^{-1} (\Sigma_p^{-1}\mu_p+\Sigma_q^{-1}\mu_q)$$

$$F=d-\mu^T \Sigma^{-1} \mu$$ $$\Rightarrow F=\mu_p^T\Sigma_p^{-1}\mu_p+\mu_q^T\Sigma_q^{-1}\mu_q-\mu^T \Sigma^{-1} \mu$$

$$E_{x\sim q(x)}[p(x)]=\frac{e^{\frac{-1}{2}F} |2\pi \Sigma|^\frac{-1}{2}}{|2\pi \Sigma_p|^\frac{-1}{2}|2\pi \Sigma_q|^\frac{-1}{2}} $$

Note that, the result is just an scalar, please let me know if you have further questions :)