Sigmoidal function in two variable

Question

I know sigmoidal functions in one variable is given by $\tfrac{1}{1+exp(-x)}$. Can anybody help me how the same will look like in two variable.

Answer 1

I would suggest reading A visual proof that neural nets can compute any function it tries to give a visual explanation of the universal approximation theorem in one and two dimension before generalizing to n dimensions, citing:

The universality theorem is well known by people who use neural networks. But why it's true is not so widely understood. Most of the explanations available are quite technical. For instance, one of the original papers proving the result did so using the Hahn-Banach theorem, the Riesz Representation theorem, and some Fourier analysis. If you're a mathematician the argument is not difficult to follow, but it's not so easy for most people. That's a pity, since the underlying reasons for universality are simple and beautiful.

In this chapter I give a simple and mostly visual explanation of the universality theorem. We'll go step by step through the underlying ideas. You'll understand why it's true that neural networks can compute any function. You'll understand some of the limitations of the result. And you'll understand how the result relates to deep neural networks.

In my case, the best of my understating comes from reading it, also you can try playing with wolfram $\frac{1}{1+e^{-x-y}}$

Answer 2

Suppose a layer in a neural net contains $n$ neurons, whose values are determined by the $m$ neurons in a previous layer. We need an activation function $\sigma(x)$, which in your case is $\frac{1}{1+\exp(-x)}$. The $i$th neuron in the $n$-neuron layer scores $y_i:=\sigma(\sum_{j=1}^mW_{ij}x_j+b_i)$, where $x_j$ is the score of the $j$th neuron in the $m$-neuron layer. All you need is the parameters $W_{ij}$ comprising a weight matrix and the parameters $b_i$ comprising a bias vector. If we say any function $f:\Bbb R\mapsto\Bbb R$ extends to acting on vectors viz. $(f(v))_i=f(v_i)$, we have the more succinct $y=\sigma(Wx+b)$. I'll leave it to you to decide whether "in two variables" means $m=2$, $n=2$ or both.