Define $F$ as the standard multi-layer feed-forward perceptron: \begin{equation} F(\mathbf{x}) = \Theta( W_1 \circ \Theta( W_2 \circ .... W_L(\mathbf{x}))) \end{equation} where $\Theta$ is the sigmoid function and $W_\ell$ is the weight matrix for layer $\ell$, every layer contains $N$ neurons, with a total of $L$ layers.
$x \in X$ is a vector space of dimension $N$.
$F$ forms a group acting on $X$.
$F$ can be parameterized by the $N \times N$ weight matrices $W_\ell$ (there are $L$ of them). The total number of parameters is $L \cdot N^2$. In other words, $F$ is parameterized by $W \in \mathbb{R}^{L N^2}$.
How can I calculate the explicit elements of the Lie algebra $\mathfrak{g} = \mbox{Lie } F$?
The sigmoid function can be either: $$\Theta(\xi) = \frac{1}{1+ e^{-a \xi}} \quad \mbox{(logistic)}$$ $$\Theta(\xi) = \tanh(\xi)$$ $$\Theta(\xi) = \begin{cases} 0, & \xi < 0 \\ \xi, & \xi \geq 0 \end{cases} \quad \mbox{(rectifier)}$$ (choose whichever one leads to an elegant solution).
I'm just beginning to learn Lie theory, and would like to apply it to neural network research. Thanks.