2-layer feed-forward neural network mapping

826 Views Asked by At

A 2-layer feed-forward neural network that takes in $x∈R^2$ and has two ReLU hidden units is defined in the figure below:

enter image description here

Note: hidden units have no offset parameters.

For illustrative purposes and for the sake of clarification, the values of the weights in the hidden layer are set such that they result in the $_1$ and $_2$ “classifiers" as shown in the in the contextual example below:

enter image description here

Here, the corresponding normal vectors are marked as (1) and (2). Given that the graph has a grid, the data points are centered thus relatively straightforward to interpret. So, if I where to approximately sketch how the input data is mapped to the 2-dimensional space of hidden unit activations $f(z_1)$ and $f(z_2)$, I get the following:

enter image description here

So far so good. Now for the part that I'm struggling with.

Based on the same 2-layer feed-forward neural network as above, the values of the weights in the hidden layer are set such that they result in the $_1$ and $_2$ “classifiers" as shown in the ($_1$,$_2$)-space in another figure below:

enter image description here

The $_1$ “classifier" with the normal $w_1=[w_{11}, w_{21}]^T$ is the line given by $z_1=x⋅w_1=0$. Similarly, the $_2$ “classifier" with the normal $w_2=[w_{12}, w_{22}]^T$ is the line given by $z_2=x⋅w_2=0$. The arrows labeled $_1$ and $_2$ point in the positive directions of the respective normal vectors. Also, the regions labeled I,II,III,IV are the 4 quadrants defined by these two lines not including the boundaries.

There is not grid, plus the data points seem to be arbitrarily scattered. Hence, my question:

What are the respective region(s) in $(_1,_2)$ space which are mapped into each of the regions in $(_1,_2)$-space, the 2-dimensional space of hidden unit activations $(_1)$ and $(_2)$?

More precisely, the region(s) in $(_1,_2)$ space which are mapped into the

  1. $_1$-axis,
  2. $_2$-axis,
  3. $(_1,_2)=0$ i.e. the origin
  4. ${(f_1,f_2):f_1>0,f_2>0}$

in $(_1,_2)$-space.

These quadrants would help me tremendously when sketching the input data in $(_1,_2)$-space.

Intuitively, I would say that region(s) in $(_1,_2)$ space are mapped into each of the following regions in $(_1,_2)$-space:

  1. $_1$-axis: I and II
  2. $_2$-axis: I and IV
  3. $(_1,_2)=0$ i.e. the origin: None
  4. ${(f_1,f_2):f_1>0,f_2>0}$: I

Some reasoning is that $_1$-axis and $_2$-axis include the origin $(f_1,f_2)=(0,0)$. Apparently, my intuition is wrong.

Regarding my question, could someone provide a rough sketch of how the $(f_1,f_2)$ space would approx. look like?

Any helping hand is highly appreciated. Many thanks!

NB: I would also like to point out that if I where given the values of each of the weights and the set of coordinates in ($_1$,$_2$)-space, I could simply replicate the neural network with a programming language and feed it the data to generate the corresponding output functions $f_1$ and $f_2$. Then, plot the values of these functions on a new graph and voila, all done. Sadly, this is not the case.