How to evaluate the integral $\int\mathbf{g}^T\mathbf{v} \exp{(-\frac{1}{2}\mathbf{v}^T \mathbf{A}\mathbf{v})}\,d\mathbf{v}$

Question

I have the following multidimensional Gaussian integral:

$$\int\textbf{g}^T\textbf{v} \exp{\left(-\frac{1}{2}\textbf{v}^T \textbf{A}\textbf{v}\right)}\,d\textbf{v},$$

where $\textbf{A}$ is a real symmetric $W\times W$ matrix, $\textbf{v} \in \mathbb{R}^W$ is a $W$-dimensional real vector and $\textbf{g}$ is some constant $W$-dimensional real vector.

Any hints on what approach would work here? Substitution perhaps?

UPDATE 1:

My problem is a sub-problem from my main task where I need to show that

$$y_{MP} = \int t\,p(t) \,dt,$$

where

$$p(t)\propto\int \exp\left(-\frac{\beta}{2}\{t-y_{MP}-\textbf{g}^T\textbf{v}\}^2-\frac{1}{2}\textbf{v}^TA\textbf{v}\right)\,d\textbf{v}.$$

$\beta, y_{MP}$ (MP stands for maximum posteriori) are constant scalars, $\textbf{g}$ is a constant vector and $A$ is a real symmetric $W\times W$matrix. The variable $t\in\mathbb{R}$ is a real number. I'm supposed to use the following facts to help me:

$$\int_{-\infty}^{\infty} \exp\left(-\frac{\lambda}{2}x^2\right)\, dx = \left(\frac{2\pi}{\lambda}\right)^{1/2}$$

$$\int_{-\infty}^{\infty} \exp\left(-\frac{1}{2}\textbf{v}^TA\textbf{v}\right)\, d\textbf{v} = (2\pi)^{W/2}|A|^{-1/2}$$

$$\int_{-\infty}^{\infty} \exp\left(-\frac{1}{2}\textbf{v}^TA\textbf{v}+\textbf{h}^T\textbf{v}\right)\, d\textbf{v} = (2\pi)^{W/2}|A|^{-1/2}\exp\left(\frac{1}{2}\textbf{h}^TA^{-1}\textbf{h}\right)$$

According to the hint, I should first integrate w.r.t $t$ and then w.r.t $\textbf{v}$. If you can help me solve my main problem, then that would be perfect answer to my question also. Thank you.

Hints from my book (page 433, tasks 10.1 and 10.2):

integrate over $t$ first, and then evaluate the integral over $\textbf{v}$ using the above three integral equations. Do the integral by expanding the square in the exponent and collect together the terms which are quadratic in $\textbf{v}$.

UPDATE 2:

Here is my own attempt and the point I'm stuck at the moment

$$\begin{aligned} \int t \, p(t)\; dt &\propto \int t \, \int \exp{\left(-\frac{\beta}{2}\left\{t-y_{MP}-\textbf{g}^T \textbf{v}\right\}^2-\frac{1}{2}\textbf{v}^T \textbf{A}\textbf{v}\right)}\; d\textbf{v}\; dt\\ &=\int \exp{\left(-\frac{1}{2}\textbf{v}^T \textbf{A}\textbf{v}\right)} \int t \exp{\left(-\frac{\beta}{2}\left\{t-y_{MP}-\textbf{g}^T \textbf{v}\right\}^2\right)}\; dt\; d\textbf{v}\\ &=\int \exp{\left(-\frac{1}{2}\textbf{v}^T \textbf{A}\textbf{v}\right)} \left[\left(y_{MP}+\textbf{g}^T \textbf{v}\right)\left(\frac{2\pi}{\beta}\right)^{1/2}\right]\; d\textbf{v} \\ &=\left(\frac{2\pi}{\beta}\right)^{1/2}\int \exp{\left(-\frac{1}{2}\textbf{v}^T \textbf{A}\textbf{v}\right)} \left(y_{MP}+\textbf{g}^T \textbf{v}\right)\; d\textbf{v} \\ &=\left(\frac{2\pi}{\beta}\right)^{1/2}\left[y_{MP}\int \exp{\left(-\frac{1}{2}\textbf{v}^T \textbf{A}\textbf{v}\right)}\; d\textbf{v} + \int \textbf{g}^T \textbf{v} \exp{\left(-\frac{1}{2}\textbf{v}^T \textbf{A}\textbf{v}\right)}\; d\textbf{v}\right] \\ &=\left(\frac{2\pi}{\beta}\right)^{1/2}\left[y_{MP}(2\pi)^{W/2}|\textbf{A}|^{-1/2} + \int \textbf{g}^T \textbf{v} \exp{\left(-\frac{1}{2}\textbf{v}^T \textbf{A}\textbf{v}\right)}\; d\textbf{v}\right] \end{aligned}$$ In the second line I simply used substitution. My biggest problem seems to be that no matter how I try to integrate w.r.t $t$ first, I always get the factor $\textbf{g}^T\textbf{v}$, which I would like to get rid of.

UPDATE 3:

I decided to post more information here from my book, maybe it can help:

In order to evaluate $p(t)$ we shall make use of the Gaussian approximation for the posterior distribution of weights (weights of a neural network), together with the distribution of network outputs. This gives:

$$p(t)\propto\int \exp\left(-\frac{\beta}{2}\{t-y(\textbf{v})\}^2\right)\exp\left(-\frac{1}{2}\textbf{v}^TA\textbf{v}\right)\,d\textbf{v},$$

where we have dropped any constant factors (i.e. factors independent of $t$. $y(\textbf{v})$ is a neural network function with network weights $\textbf{v}$. In addition, we shall assume that the width of the posterior distribution (determined by the Hessian matrix $\textbf{A}$ of the error function) is sufficiently narrow that we may approximate the network function $y(\textbf{v})$ by its linear expansion around $\textbf{v}_{MP}$

$$y(\textbf{v}) = y(\textbf{v}_{MP}) + \textbf{g}^T\textbf{v},$$

where $\textbf{g} \equiv \nabla y(\textbf{v})|_{\textbf{v}=\textbf{v}_{MP}}$. $y_{MP}\equiv y(\textbf{v}_{MP})$

Answer 1

I think I got my main problem solved perhaps. Two things that helped me:

1. Group the terms as suggested in the hint by author
2. Integrate w.r.t to $\textbf{v}$ first instead of $t$

here goes:

$$\begin{aligned} \int t \, p(t)\; dt &\propto \int t \, \int \exp{\left(-\frac{\beta}{2}\left\{t-y_{MP}-\textbf{g}^T \textbf{v}\right\}^2-\frac{1}{2}\textbf{v}^T \textbf{A}\textbf{v}\right)}\; d\textbf{v}\; dt\\ &=\int t\int \exp\left(-\frac{\beta}{2}\{t^2-2ty_{MP}-2t\textbf{g}^T\textbf{v}+2y_{MP}\textbf{g}^T\textbf{v}+y_{MP}^2 + \textbf{v}^T\textbf{g}\textbf{g}^T\textbf{v}\}-\frac{1}{2}\textbf{v}^T\textbf{A}\textbf{v}\right)\;d\textbf{v}\;dt\\ &=\int t\int \exp\left(-\frac{\beta}{2}\{t-y_{MP}\}^2+\beta (t-y_{MP})\textbf{g}^T\textbf{v} -\frac{\beta}{2} \textbf{v}^T\textbf{g}\textbf{g}^T\textbf{v}-\frac{1}{2}\textbf{v}^T\textbf{A}\textbf{v}\right)\;d\textbf{v}\;dt\\ &=\int t\exp\left(-\frac{\beta}{2}\{t-y_{MP}\}^2\right)\int \exp\left(-\frac{1}{2}\textbf{v}^T\left(\textbf{A}+\beta\textbf{g}\textbf{g}^T\right)\textbf{v}+\beta (t-y_{MP})\textbf{g}^T\textbf{v}\right)\;d\textbf{v}\;dt\\ &=\int t\exp\left(-\frac{\beta}{2}\{t-y_{MP}\}^2\right) \left[(2\pi)^{W/2}|\textbf{A}+\beta\textbf{g}\textbf{g}^T|^{-1/2} \exp\left(\frac{1}{2}\beta (t-y_{MP})\textbf{g}^T\left(\textbf{A}+\beta\textbf{g}\textbf{g}^T\right)^{-1}\beta (t-y_{MP})\textbf{g}\right)\right]\;dt. \end{aligned}$$

I will next denote $\textbf{A}+\beta\textbf{g}\textbf{g}^T=\textbf{C}$ and $\textbf{g}^T \textbf{C}^{-1}\textbf{g} = D$ so I get:

$$\begin{aligned} \int t \, p(t)\; dt &\propto \int t\exp\left(-\frac{\beta}{2}\{t-y_{MP}\}^2\right) \left[(2\pi)^{W/2}|\textbf{C}|^{-1/2} \exp\left(\frac{1}{2}\beta^2 \{t-y_{MP}\}^2 D\right)\right]\;dt\\ &= (2\pi)^{W/2}|\textbf{C}|^{-1/2} \int t\exp\left(-\frac{\beta}{2}\{t-y_{MP}\}^2\right) \exp\left((-D\beta)\left(-\frac{\beta}{2}\right) \{t-y_{MP})^2 \right)\;dt\\ &= (2\pi)^{W/2}|\textbf{C}|^{-1/2} \int t \exp\left((1-D\beta)\left(-\frac{\beta}{2}\right) \{t-y_{MP})^2 \right)\;dt\\ &= (2\pi)^{W/2}|\textbf{C}|^{-1/2} \int t \exp\left(-\frac{\lambda}{2} \{t-y_{MP})^2 \right)\;dt, \end{aligned}$$

where $\lambda = \beta-D\beta^2$. Now it down to the last integral. I will use substitution twice. First I set $u=\exp\left(-\frac{\lambda}{2}\{t-y_{MP}\}^2\right)$, from which I get

$$\int du =-\lambda\int t \exp\left(-\frac{\lambda}{2}\{t-y_{MP}\}^2\right)\;dt+\lambda y_{MP}\int \exp\left(-\frac{\lambda}{2}\{t-y_{MP}\}^2\right)\;dt$$

or

$$\begin{aligned} \int t \exp\left(-\frac{\lambda}{2}\{t-y_{MP}\}^2\right)\;dt &= -\frac{1}{\lambda}\int du + y_{MP}\int \exp\left(-\frac{\lambda}{2}\{t-y_{MP}\}^2\right)\;dt\\ &= -\frac{1}{\lambda}\left[\exp\left(-\frac{\lambda}{2}\{t-y_{MP}\}^2\right)\right]^{\infty}_{-\infty} + y_{MP}\int \exp\left(-\frac{\lambda}{2}\{t-y_{MP}\}^2\right)\;dt\\ &= 0 + y_{MP}\int_{-\infty}^{\infty} \exp\left(-\frac{\lambda}{2}s^2\right)\;ds\\ &= y_{MP}\left(\frac{2\pi}{\lambda}\right)^{1/2}. \end{aligned}$$

On the previous I used substitution $s=t-y_{MP}$. So now, if I have done everything correctly I have that the main integral is:

$$\begin{aligned} \int t \, p(t)\; dt &\propto y_{MP}\left(\frac{2\pi}{\lambda}\right)^{1/2}(2\pi)^{W/2}|\textbf{C}|^{-1/2}\\ &= y_{MP}\left(\left(\frac{2\pi}{\beta-\beta^2\textbf{g}^T \left(\textbf{A}+\beta\textbf{g}\textbf{g}^T\right)^{-1}\textbf{g}}\right)^{1/2} (2\pi)^{W/2}|\textbf{A}+\beta\textbf{g}\textbf{g}^T|^{-1/2}\right) \end{aligned}.$$

This seems to be correct by looking at the results of the book. This isn't given explicitly, but in the book I use the author states that the corresponding variance (hinting I got something right) for $p(t)$ is:

$$\sigma_t^2 = \frac{1}{\beta-\beta^2\textbf{g}^T \left(\textbf{A}+\beta\textbf{g}\textbf{g}^T\right)^{-1}\textbf{g}}=\frac{1}{\lambda}.$$

As a follow up in the same problem, I needed to show that:

$$\sigma^2_t = \frac{1}{\lambda}=\frac1\beta+\textbf{g}^T\textbf{A}^{-1}\textbf{g},$$

which indeed is the case:

\begin{aligned} \sigma^2_t &= \frac{1}{\beta-\beta^2\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}}\\ &= \frac{1}{\beta-\beta^2\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}}\times\frac{\textbf{g}^T(\textbf{I}+\beta\textbf{A}^{-1}\textbf{g}\textbf{g}^T)\textbf{g}}{\textbf{g}^T(\textbf{I}+\beta\textbf{A}^{-1}\textbf{g}\textbf{g}^T)\textbf{g}}\\ &= \frac{\textbf{g}^T\textbf{g}+\textbf{g}^T\beta\textbf{A}^{-1}\textbf{g}\textbf{g}^T\textbf{g}}{\beta\textbf{g}^T\textbf{g}+\beta^2\textbf{g}^T\textbf{A}^{-1}\textbf{g}\textbf{g}^T\textbf{g}-\beta^2\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}\textbf{g}^T\textbf{g}-\beta^2\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}\textbf{g}^T(\beta\textbf{A}^{-1}\textbf{g}\textbf{g}^T)\textbf{g}}\\ &=\frac{1+\textbf{g}^T\beta\textbf{A}^{-1}\textbf{g}}{\beta+\beta^2\textbf{g}^T\textbf{A}^{-1}\textbf{g}-\beta^2\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}-\beta^2\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}\textbf{g}^T\beta\textbf{A}^{-1}\textbf{g}}\\ &=\frac{\frac1\beta+\textbf{g}^T\textbf{A}^{-1}\textbf{g}}{1+\beta\textbf{g}^T\textbf{A}^{-1}\textbf{g}-\beta\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}-\beta\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}\textbf{g}^T\beta\textbf{A}^{-1}\textbf{g}}, \end{aligned}

so now we must have:

$$\beta\textbf{g}^T\textbf{A}^{-1}\textbf{g}-\beta\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}-\beta\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}\textbf{g}^T\beta\textbf{A}^{-1}\textbf{g}=0$$

$$\textbf{g}^T\textbf{A}^{-1}\textbf{g}-\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}-\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}\textbf{g}^T\beta\textbf{A}^{-1}\textbf{g}=0$$

$$\textbf{g}^T\textbf{A}^{-1}\textbf{g}-\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}=\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}\textbf{g}^T\beta\textbf{A}^{-1}\textbf{g}$$

Next, I will denote: $\textbf{C}=(\textbf{A}+\beta\textbf{g}\textbf{g}^T)$

$$\textbf{g}^T\textbf{A}^{-1}\textbf{g}-\textbf{g}^T\textbf{C}^{-1}\textbf{g}=\beta\textbf{g}^T\textbf{C}^{-1}\textbf{g}\textbf{g}^T\textbf{A}^{-1}\textbf{g}$$

$$\textbf{g}^T\textbf{A}^{-1}\textbf{g}\textbf{g}^T(\textbf{g}\textbf{g}^T)^{-1}\textbf{C}-\textbf{g}^T\textbf{C}^{-1}\textbf{g}\textbf{g}^T(\textbf{g}\textbf{g}^T)^{-1}\textbf{C}=\beta\textbf{g}^T\textbf{C}^{-1}\textbf{g}\textbf{g}^T\textbf{A}^{-1}\textbf{g}\textbf{g}^T(\textbf{g}\textbf{g}^T)^{-1}\textbf{C}$$

$$\textbf{g}^T\textbf{A}^{-1}\textbf{C}-\textbf{g}^T=\beta\textbf{g}^T\textbf{C}^{-1}\textbf{g}\textbf{g}^T\textbf{A}^{-1}\textbf{C}$$

$$\textbf{g}\textbf{g}^T\textbf{A}^{-1}\textbf{C}-\textbf{g}\textbf{g}^T=\beta\textbf{g}\textbf{g}^T\textbf{C}^{-1}\textbf{g}\textbf{g}^T\textbf{A}^{-1}\textbf{C}$$

$$\textbf{C}(\textbf{g}\textbf{g}^T)^{-1}\textbf{g}\textbf{g}^T\textbf{A}^{-1}\textbf{C}-\textbf{C}(\textbf{g}\textbf{g}^T)^{-1}\textbf{g}\textbf{g}^T=\beta\textbf{C}(\textbf{g}\textbf{g}^T)^{-1}\textbf{g}\textbf{g}^T\textbf{C}^{-1}\textbf{g}\textbf{g}^T\textbf{A}^{-1}\textbf{C}$$

$$\textbf{C}\textbf{A}^{-1}\textbf{C}-\textbf{C}=\beta\textbf{g}\textbf{g}^T\textbf{A}^{-1}\textbf{C}$$

$$(\textbf{A}+\beta\textbf{g}\textbf{g}^T)\textbf{A}^{-1}(\textbf{A}+\beta\textbf{g}\textbf{g}^T)-(\textbf{A}+\beta\textbf{g}\textbf{g}^T)=\beta\textbf{g}\textbf{g}^T\textbf{A}^{-1}(\textbf{A}+\beta\textbf{g}\textbf{g}^T)$$

$$(\textbf{A}+\beta\textbf{g}\textbf{g}^T)(\textbf{I}+\beta\textbf{A}^{-1}\textbf{g}\textbf{g}^T)-(\textbf{A}+\beta\textbf{g}\textbf{g}^T)=\beta\textbf{g}\textbf{g}^T\textbf{A}^{-1}(\textbf{A}+\beta\textbf{g}\textbf{g}^T)$$

$$(\textbf{A}+\beta\textbf{g}\textbf{g}^T)(\textbf{I}+\beta\textbf{A}^{-1}\textbf{g}\textbf{g}^T)-(\textbf{A}+\beta\textbf{g}\textbf{g}^T)=\beta(\textbf{g}\textbf{g}^T+\beta\textbf{g}\textbf{g}^T\textbf{A}^{-1}\textbf{g}\textbf{g}^T)$$

$$(\textbf{A}+\beta\textbf{g}\textbf{g}^T)(\beta\textbf{A}^{-1}\textbf{g}\textbf{g}^T)=\beta(\textbf{g}\textbf{g}^T+\beta\textbf{g}\textbf{g}^T\textbf{A}^{-1}\textbf{g}\textbf{g}^T)$$

$$\beta(\textbf{g}\textbf{g}^T+\beta\textbf{g}\textbf{g}^T\textbf{A}^{-1}\textbf{g}\textbf{g}^T)=\beta(\textbf{g}\textbf{g}^T+\beta\textbf{g}\textbf{g}^T\textbf{A}^{-1}\textbf{g}\textbf{g}^T).$$

In other words,

$$\sigma^2_t = \frac{1}{\beta-\beta^2\textbf{g}^T(\textbf{A}+\beta\textbf{g}\textbf{g}^T)^{-1}\textbf{g}}=\frac1\beta+\textbf{g}^T\textbf{A}^{-1}\textbf{g}.$$

Another update:

I tried also to compute the integral :

$$ \int_{-\infty}^{\infty} \, \int_{\mathbb{R}^W} \exp{\left(-\frac{\beta}{2}\left\{t-y_{MP}-\textbf{g}^T \textbf{v}\right\}^2-\frac{1}{2}\textbf{v}^T \textbf{A}\textbf{v}\right)}\; d\textbf{v}\; dt,$$

first w.r.t $\textbf{v}$ and then w.r.t $t$ and then vice versa to check that both ways result in the same outcome. I was confused at start because I got:

$$\begin{aligned} I_{\textbf{v}, t}&=\int_{-\infty}^{\infty} \, \int_{\mathbb{R}^W} \exp{\left(-\frac{\beta}{2}\left\{t-y_{MP}-\textbf{g}^T \textbf{v}\right\}^2-\frac{1}{2}\textbf{v}^T \textbf{A}\textbf{v}\right)}\; d\textbf{v}\; dt\\ &= (2\pi)^{W/2}(2\pi\sigma^2_t)^{1/2}|\textbf{A}+\beta\textbf{g}\textbf{g}^T|^{-1/2} \end{aligned},$$

and:

$$\begin{aligned} I_{t, \textbf{v}}&=\int_{\mathbb{R}^W}\,\int_{-\infty}^{\infty} \exp{\left(-\frac{\beta}{2}\left\{t-y_{MP}-\textbf{g}^T \textbf{v}\right\}^2-\frac{1}{2}\textbf{v}^T \textbf{A}\textbf{v}\right)}\;dt\; d\textbf{v}\\ &= (2\pi)^{W/2}\left(\frac{2\pi}{\beta}\right)^{1/2}|\textbf{A}|^{-1/2} \end{aligned}.$$

So now $I_{t, \textbf{v}}$ should equal $I_{\textbf{v}, t}$:

$$(2\pi)^{W/2}\left(\frac{2\pi}{\beta}\right)^{1/2}|\textbf{A}|^{-1/2}=(2\pi)^{W/2}(2\pi\sigma^2_t)^{1/2}|\textbf{A}+\beta\textbf{g}\textbf{g}^T|^{-1/2}$$

$$\left(\frac{2\pi}{\beta}\right)^{1/2}|\textbf{A}|^{-1/2}=(2\pi\sigma^2_t)^{1/2}|\textbf{A}+\beta\textbf{g}\textbf{g}^T|^{-1/2}$$

$$\left(\frac{2\pi}{\beta}\right)^{1/2}|\textbf{A}|^{-1/2}=\left[2\pi\left(\frac{1}{\beta}+\textbf{g}^T\textbf{A}^{-1}\textbf{g}\right)\right]^{1/2}|\textbf{A}+\beta\textbf{g}\textbf{g}^T|^{-1/2},$$

next I will square both sides:

$$\frac{2\pi}{\beta|\textbf{A}|}=\frac{2\pi\left(\frac{1}{\beta}+\textbf{g}^T\textbf{A}^{-1}\textbf{g}\right)}{|\textbf{A}+\beta\textbf{g}\textbf{g}^T|}$$

$$\frac{1}{|\textbf{A}|}=\frac{1+\beta\textbf{g}^T\textbf{A}^{-1}\textbf{g}}{|\textbf{A}+\beta\textbf{g}\textbf{g}^T|}.$$

This is where I thought something had gone wrong until I found out about the matrix determinant lemma making:

$$|\textbf{A}+\beta\textbf{g}\textbf{g}^T|=(1+\beta\textbf{g}^T\textbf{A}^{-1}\textbf{g})|\textbf{A}|,$$

that is:

$$\frac{1}{|\textbf{A}|}=\frac{1+\beta\textbf{g}^T\textbf{A}^{-1}\textbf{g}}{(1+\beta\textbf{g}^T\textbf{A}^{-1}\textbf{g})|\textbf{A}|}$$

$$\frac{1}{|\textbf{A}|}=\frac{1}{|\textbf{A}|},$$

so indeed $I_{t, \textbf{v}}=I_{\textbf{v}, t}\;\;\;\blacksquare$