Integral of softmax (i.e multi-variate sigmoid) over hyper-cube

1.1k Views Asked by At

Let $n$ and $k$ be a positive integers and $\mathbf{b}_1,\ldots,\mathbf{b}_k \in \mathbb R^n$ (with $\mathbf{b}_l \ne 0$ for at least one $l$), $c_1,\ldots,c_k \in \mathbb R$. What does the following integral

$$I(\mathbf{b}_1,\ldots,\mathbf{b}_k,c_1,\ldots,c_k): =\int_{[0,1]^n}\frac{1}{1 + \sum_{l=1}^k\exp(\mathbf{x}^T\mathbf{b}_l + c_l)}d \mathbf{x}$$

evaluate to ?

Observations

The 1-dimensional binary case (i.e $n=k=1$) is trivial. Indeed, from $$\int \frac{1}{1 + \exp(-x)}dx = \ln (1 + \exp(x)) + \text{constant}, $$

one gets $$ \begin{split} I(b, c) &= \int_0^1 \frac{1}{1 + \exp(bx + c)}dx = -\frac{1}{b} \int_{-c}^{-b-c} \frac{1}{1 + \exp(-z)}dz \\ &= \frac{1}{b}\ln\left(\frac{1 + \exp(-c)}{1 + \exp(-b-c)}\right), \end{split} $$ where we've used the change of variable: $-z = bx + c$. In particular, one has

$$ I(-1,0) = \int_0^1 \frac{1}{1 + \exp(-x)}dx = \ln\left(\frac{1 + e}{2}\right) \approx 0.62$$

1

There are 1 best solutions below

0
On

Solution to multi-dimensional binary case (i.e $n \ge k = 1$)

First ingredient. One has the following Macluarin expansion (e.g see this page) $$\frac{1}{1 + \exp(-z)} = \sum_{m=0}^\infty \frac{(-1)^{m+1}(2^{m+1} - 1)B_{m+1}}{m + 1}z^m,$$ where $B_m$ is the $mth Bernoulli number.

Second ingredient. Recall the multinomial binomial theorem

$$ \left(\sum_{i=1}^n z_i\right)^m = \sum_{|k| = m}\frac{m!}{k_1!\ldots k_n!}\prod_{i=1}^n z_i^{k_i}. $$

Third ingredient. If $k_1,\ldots k_n$ are real numbers all $\ne - 1$, then $$\int_{[0,1]^n}\prod_{i=1}^n z_i^{k_i}d\mathbf{z} = \frac{1}{\prod_{i=1}^n k_i + 1}. $$

Putting things together gives

$$ \begin{split} \int_{[0,1]^n}\frac{1}{1 + \exp(-\mathbf{x}^T\mathbf{b} - b_0)}d\mathbf{x}&= \sum_{m=0}^\infty \frac{(-1)^{m+1}(2^{m+1} - 1)B_{m+1}}{m + 1}\sum_{|k| = m}\frac{m!}{k_0!k_1! \ldots k_n!}b_0^{k_{0}}\prod_{i=1}^n \frac{1}{k_i + 1}b_i^{k_i}\\ &= \sum_{m=0}^\infty \frac{(-1)^{m+1}(2^{m+1} - 1)B_{m+1}}{m + 1}\sum_{|k| = m}\frac{m!}{k_0!(k_1 + 1)!\ldots (k_n + 1)!}\prod_{i=0}^{n} b_i^{k_i} \end{split} $$

Disappointment

However (assuming I haven't made a mistake somewhere down the line), the above formula is rather cumbersome and dirty. Maybe someone here knows how to rewrite this more compactly using logarithms and exponentials ?