I have normally distibuted vector $X$ with known variance $Var(X)$ and mean $\overline X$ and then I take new vector $Y$ as $W*X$ where $W$ is normally distributed matrix. If I required $Y$ to have same variance and mean as $X$, what should $Var(W)$ and $\overline W$ be ? More generally - given $\overline X$, $Var(X)$, $\overline W$ and $Var(W)$, what would $\overline {W*X}$ and $Var(W*X)$ be ?
Note : For the desired application ( initialization in neural networks), the equality of means isn't really necessary as both means are likely to be almost zero anyway
So $X$ has i.i.d. normal entries, and $W$ has i.i.d. normal entries.
The entries of $WX$ will not be normal.
But you can answer your question about means and variances without any normality assumptions on $W$ and $X$.
Let $W$ have dimension $m \times n$.
Since $$Y_i = \sum_{j=1}^n W_{i,j} X_j$$ we can use independence to obtain $$E[Y_i] = \sum_{j=1}^n E[W_{i,j}] E[X_j] = n \overline{W} \overline{X}$$ and $$\text{Var}(Y_i) = \sum_{j=1}^n \text{Var}(W_{i,j} X_j) = n[ \text{Var}(X) \text{Var}(W) + \text{Var}(X) \overline{W}^2 + \text{Var}(W) \overline{X}^2 ].$$
Setting these equal to $\overline{X}$ and $\text{Var}(X)$ respectively yields $\overline{W} = 1/n$ and $$\text{Var}(W) = \frac{\left(\frac{1}{n} - \frac{1}{n^2}\right)\text{Var}(X)}{\text{Var}(X) + \overline{X}^2}$$