How many parameters does the neural network have?

Question

We have a neural network with an input layer of ℎ0 nodes, hidden layers of ℎ1 , ℎ2 , ℎ3 , ..., ℎ−1 nodes respectively and an output layer of ℎ nodes. How many parameters does the network have?

Answer 1

Suppose the network has $784$ inputs, $16$ nodes in $2$ hidden layers and $10$ nodes in the output layer.

The amount of parameters (meaning weights and bias that make up the cost function) is then:

For the weights: $$784\times 16+16\times16+16\times10=12960$$

For the bias components:

We have $32$ neurons in the hidden layers and $10$ in the output, so we have $$32+10 = 42$$ biases.

So in total, the amount of parameters in this neural network is $13002$.

Answer 2

for simplicity lets say the $l=5$ clearly we have,

           i=h0        //inputs
           h=[h1,h2,h3,h4]   //hidden layers
           o=h5             //output

therefore the no of params $= (h0*h1 + h1*h2 + h2*h3 + h4*h5) + [h1+h2+h3+h4+h5]$ =click here to view the expression

Answer 3

Let us say we have $l$ layers in a feed forward neural network, numbered from $0$ to $l-1$, with layer $0$ being input layer (no bias terms here as there are no neurons in this layer, they are simply input nodes). Let number of nodes in each layer be $n_0,n_1,n_2,n_3\dots n_{l-1}$. Then the number of parameters (weights) of the network including biases will be $$n_0\times n_1 + n_1\times n_2 + n_2\times n_3 + \dots + n_{l-2}\times n_{l-1} + ( n_1+n_2+n_3+\dots n_{l-1})\ .$$ The last expression (in bracket) is the number of bias terms in the network. This equals to the actual number of neurons in the feed forward network, as each neuron is having one bias input. Please note that giving bias input is not compulsory, although many libraries adds these inputs(typically fixed at 1) automatically to each neuron. It is important to note that input nodes are not neurons.

The example above is a 2x2x2 network. according to the formula the number of model parameters(weights) of this Neural Network model = (2x2)+(2x2)+(2+2)=12.