I have been self-studying deep learning, and encountered the question below from Michael Nielsen's deep learning online book:
Identity neuron: Consider a neuron with a single input, $x$, a corresponding weight, $w_1$, a bias $b$, and a weight $w_2$ on the output. Show that by choosing the weights and bias appropriately, we can ensure $w_2\sigma(w_1x+b)\approx x$ for $x\in[0,1]$. Such a neuron can thus be used as a kind of identity neuron, that is, a neuron whose output is the same (up to rescaling by a weight factor) as its input. Hint: It helps to rewrite $x=\frac{1}{2}+\Delta$, to assume $w_1$ is small, and to use a Taylor series expansion in $w_1\Delta$.
Taylor's series is $\sum_0^\infty \frac{f^{(n)}(a)}{n!}(x-a)^n$. And so $f(x) = w_2\sigma(w_1x+b)- x = \sum_0^\infty \frac{f^{(n)}(a)}{n!}(x-a)^n$.
Given the above problem, what kind of value am I supposed to set for $a$? Maybe, $\frac{1}{2}+\Delta$? And $\sigma(x) = \frac{1}{1+exp(-x)}, \sigma'(x) = \sigma(x)(1-\sigma(x)), \sigma''(x) = \sigma(x)(1-\sigma(x))(1-2\sigma(x)), etc$ . Until how many derivatives am I supposed to compute? And how can I lead the infinite series to a specific equation? I am new to Taylor's series so I have no idea how to actually apply Taylor's series to a real question. Any insight, please?