Suppose we have $$f(g) = \log\left(\sum_{i = 1}^N \begin{cases} |a_i| e^{g_i}, & a_i \geq 0 \\ |a_i| (2 - e^{g_i}), & a_i < 0 \end{cases} \right) $$ where $g = (g_1, \ldots, g_N)$, each $g_i$ is independently $\mathcal{N}(\mu_i, \sigma_i^2)$, and $\sum_{i = 1}^N |a_i| = 1$.
I want to calculate the mean and variance of $f(g)$.
For my practical purposes, each $\mu_i$ and $\sigma_i$ are relatively close to zero, so I have attempted a solution by taking a second order Taylor-expansion of $f$ around 0.
\begin{align} f(g) &= \log\left(\sum_{i = 1}^N \begin{cases} |a_i| (1 + g_i + \frac{g_i^2}{2} + O(g_i^3)), & a_i \geq 0 \\ |a_i| (1 - g_i - \frac{g_i^2}{2} + O(g_i^3)), & a_i < 0 \end{cases} \right) \\ &= \log\left(\sum_{i = 1}^N |a_i| + a_i g_i + \frac{a_i g_i^2}{2} + O(g_i^3)) \right) \\ &= \log\left(1 + \sum_{i = 1}^N a_i g_i + \frac{a_i g_i^2}{2} + O(g_i^3)) \right) \quad \text{since } \sum_{i = 1}^N |a_i| = 1 \\ &= \left(\sum_{i=1}^N a_i g_i + \frac{a_ig_i^2}{2}\right) - \frac{\left(\sum_{i=1}^N a_i g_i + \frac{a_ig_i^2}{2} + O(g_i^3) \right)^2}{2} + \sum_{i=1}^N O(g_i^3) \\ &= \left(\sum_{i=1}^N a_i g_i + \frac{a_ig_i^2}{2}\right) - \frac{\left(\sum_{i=1}^N a_i g_i\right)^2}{2} + \sum_{i=1}^N O(g_i^3) \end{align}
so my approximation is $$\tilde{f}(g) = \left(\sum_{i=1}^N a_i g_i + \frac{a_ig_i^2}{2}\right) - \frac{\left(\sum_{i=1}^N a_i g_i\right)^2}{2}$$ and \begin{align} \mathbb{E}[\tilde{f}(g)] &= \sum_{i=1}^N \left(a_i \mu_i + \frac{a_i (\mu_i^2 + \sigma_i^2)}{2}\right) - \frac{\left(\sum_{i=1}^N a_i \mu_i\right)^2 + \left(\sum_{i=1}^N a_i^2 \sigma_i^2\right)}{2}\\ Var(\tilde{f}(g)) &= \ ? \end{align}
My calculations start getting very messy very quickly once I try to compute $Var(\tilde{f}(g))$, and furthermore, I am not so sure whether these approximations are kosher to begin with. Am I making assumptions about higher order moments of each $g_i$ that I am not justifying? I am implicitly guessing that I need to keep the quadratic terms of my Taylor expansions to have a reasonable estimate of the second moment of $f(g)$, is this correct?
Thank you for the help!
In general, we can use a multivariate Taylor expansion of the whole function to compute the expectation and the variance (rather than expanding the exponential terms and then the logarithm).
The second order expansion around some (column) vector $g'$ is
$$f(g) \approx f(g') + \nabla f(g')^T(g-g') + \frac{1}{2} (g-g')^T \mathbf{H}_f(g') (g-g')$$
where $\mathbf{H}_f(g')$ is the Hessian of $f$ at $g'$.
The expectation can be approximated by our second order expansion about the mean ($g' = \mu$):
\begin{align} \mathbb{E}[f(g)] &\approx \mathbb{E} \left[ f(\mu) + \nabla f(\mu)^T(g-\mu) + \frac{1}{2} (g-\mu)^T \mathbf{H}_f(\mu) (g-\mu) \right]\\ &= f(\mu) + \nabla f(\mu)^T\mathbb{E} \left[ (g-\mu) \right] + \frac{1}{2} \mathbb{E} \left[ (g-\mu)^T \mathbf{H}_f(\mu) (g-\mu) \right]\\ &= f(\mu) + \frac{1}{2} Tr\left(\mathbf{H}_f(\mu) \Lambda \right)\\ \end{align} where $\Lambda$ is the covariance matrix of $g$.
Now, if we want to compute $\mathbb{E}[f(g)^2]$, we can use the expansion of $s(g) = f(g^2)$ around $g'$, where:
$$\nabla s(g') = 2f(g') \nabla f(g')$$ and $$\mathbf{H}_s(g') = 2f(g')\mathbf{H}_f(g') + 2\nabla f(g') \nabla f(g')^T$$
Then the expectation of $f(g)^2$ can be approximated by another second-order Taylor series centered at $\mu$:
\begin{align} \mathbb{E}[f(g)^2] &\approx \mathbb{E} \left[ s(\mu) + \nabla s(\mu)^T(g-\mu) + \frac{1}{2} (g-\mu)^T \mathbf{H}_s(\mu) (g-\mu) \right]\\ &= f(\mu)^2 + 2f(\mu)\nabla f(\mu)^T\mathbb{E} \left[ (g-\mu) \right] + f(\mu) \mathbb{E} \left[ (g-\mu)^T \mathbf{H}_f(\mu) (g-\mu) \right] + \mathbb{E} \left[ (g-\mu)^T \nabla f(\mu) \nabla f(\mu)^T (g-\mu) \right]\\ &= f(\mu)^2 + f(\mu) \mathbb{E} \left[ (g-\mu)^T \mathbf{H}_f(\mu) (g-\mu) \right] + \mathbb{E} \left[ (g-\mu)^T \nabla f(\mu) \nabla f(\mu)^T (g-\mu) \right]\\ &= f(\mu)^2 + f(\mu)Tr\left(\mathbf{H}_f(\mu) \Lambda \right) + Tr\left(\nabla f(\mu) \nabla f(\mu)^T \Lambda \right) \end{align}
Finally, \begin{align} Var(f(g)) &= \mathbb{E}[f(g)^2] - \mathbb{E}[f(g)]^2\\ &\approx f(\mu)^2 + f(\mu)Tr\left(\mathbf{H}_f(\mu) \Lambda \right) + Tr\left(\nabla f(\mu) \nabla f(\mu)^T \Lambda \right) - \left(f(\mu) + \frac{1}{2} Tr\left(\mathbf{H}_f(\mu) \Lambda \right)\right)^2\\ &= Tr\left(\nabla f(\mu) \nabla f(\mu)^T \Lambda \right) - \frac{1}{4} \left( Tr\left(\mathbf{H}_f(\mu) \Lambda \right)\right)^2 \end{align}
We can also compute these approximations around $g' = 0$ as I originally asked, instead of at $g' = \mu$.
$$\mathbb{E}[f(g)] \approx \mathbb{E}[f(0) + \nabla f(0)^T(g-0) + \frac{1}{2} (g-0)^T \mathbf{H}_f(0) (g-0)]$$
We have:
$$f(0) = \log\left(\sum_{i=1}^N a_i\right) = 0$$
$$\nabla f(0)_i = a_i$$
$$\mathbf{H}_f(0)_{ij} = a_i(\delta_{ij} - a_j)$$
So:
\begin{align} \mathbb{E}[f(g)] &\approx \mathbb{E}[a^Tg + \frac{1}{2} g^T \mathbf{H}_f(0)g)] \\ &= a^T\mu + \frac{1}{2} Tr(\mathbf{H}_f(0)\Lambda) + \frac{1}{2} \mu^T\mathbf{H}_f(0)\mu \end{align}
And: \begin{align} \mathbb{E}[f(g)^2] &\approx \mathbb{E} \left[ s(0) + \nabla s(0)^T(g-0) + \frac{1}{2} (g-0)^T \mathbf{H}_s(0) (g-0) \right]\\ &= f(0)^2 + 2f(0)\nabla f(0)^T\mathbb{E} \left[ g \right] + f(0) \mathbb{E} \left[ g^T \mathbf{H}_f(0) g \right] + \mathbb{E} \left[ g^T \nabla f(0) \nabla f(0)^T g \right]\\ &= \mathbb{E} \left[ g^T \nabla f(0) \nabla f(0)^T g \right]\\ &= Tr\left( \nabla f(0) \nabla f(0)^T \Lambda\right) + \mu^T \nabla f(0) \nabla f(0)^T \mu\\ &= Tr\left( \nabla f(0) \nabla f(0)^T \Lambda\right) + (a^T\mu)^2\\ \end{align}