Reproduce a 0-mean, unit-variance normal from weighted average of samples

Question

I'm training a neural network & my architecture's stability depends on each layer returning samples from a 0-mean, unit-variance normal. Some layers need to combine $V$ vectors (a, b, c) with learnable weights $W$. I naively implemented this with a weighted average but empirically found the variance of my output distribution is all wrong! ...and depends on each value in $W$, even given $\sum W = 1$. Given $W$, can we figure out some scalar $N$ to multiply our output samples by to fix the variance?

mean = 0
variance = 1
samples = 100000

# vectors
a = np.random.normal(mean, variance, samples)
b = np.random.normal(mean, variance, samples)
c = np.random.normal(mean, variance, samples)

# weights
weight_a = 0.3
weight_b = 0.3
weight_c = 0.4

result = (a * weight_a) + (b * weight_b) + (c * weight_c)
print(np.mean(result), np.var(result))
# 0.001295722463321136 0.34123206614565454

NB: the real network will use 16-1024 vectors with 16-256 values each

Apologies for writing the example in code rather than math, this stuff is very unfamiliar territory for me.

Thanks for helping!

Answer 1

If each of $A$, $B$, and $C$ are independent standard normal (mean $0$, variance $1$) random variables, and you want some weighted average of these to also be standard normal, then note $$\operatorname{E}[w_A A + w_B B + w_C C] = w_A \operatorname{E}[A] + w_B \operatorname{E}[B] + w_C \operatorname{E}[C] = 0$$ by linearity of expectation, but $$ \operatorname{Var}[w_A A + w_B B + w_C C] = w_A^2 \operatorname{Var}[A] + w_B^2 \operatorname{Var}[B] + w_C^2 \operatorname{Var}[C] = w_A^2 + w_B^2 + w_C^2.$$ So the correct normalization factor needs to be $$\frac{w_A A + w_B B + w_C C}{w_A^2 + w_B^2 + w_C^2}.$$ It is worth noting that in the above, we don't need the sum of the weights to be unity: the zero mean is assured by the fact that the individual means are zero, and the unit variance will arise for any choice of weights.