Bounding the Variance of a Product of Dependent Random Variables

Question

I want to bound from above the variance of the following function: $$g(w) = \|\sigma'(\langle z, w \rangle )w\|^2$$ where $w \in R^d$ is a vector of $d$ i.i.d random variables $w_i$ of normal distribution, with mean $0$, and variance $\frac{1}{d}$ as entries, and $z \in R^d$ is a constant, which makes $\langle z, w \rangle$ a weighted sum of Gaussians. $\sigma$ could be any sigmoid-like function, it's smooth and I think I can assume it is bounded, and so is its derivative (of course I'd like my results to be as general as possible). $$ $$ Knowing that $Var(\|w\|^2) = d Var(w_i^2) = d \frac{2}{d^2} = \frac{2}{d}$, I've tried looking at $g(w)$ as the product of $\|w\|^2$ and $\sigma'(\langle z,w \rangle)^2$ which is obviously a product of two dependent random variables, and that has made the whole thing a bit of a mess for me. I see that sigmoid-like functions are somewhat degenerated, which works in my favor, and yet I am still stuck. I'd love your help on this. Thanks!!

Answer 1

Did you mean to upper bound $\mathbb{E}[g(w)]$?

If so, as you said, you can assume that $\|\sigma'(\cdot)\| \leq B$ for some $B$, and then: \begin{align} \|\sigma'(\langle w, z \rangle) w\|^2 \leq \|\sigma'(\langle w, z \rangle)\|^2 \cdot\| w\|^2 \leq B^2\|w\|^2 \end{align} and thus \begin{align} \mathbb{E}[g(w)] \leq B^2\mathbb{E}[\|w\|^2] = B^2. \end{align}

If you want to bound the variance, then: \begin{align} Var(g(w)) \leq \mathbb{E}[g^2(w)] \leq B^4\mathbb{E}[\|w\|^4] = 3B^4/d. \end{align}