What kind of neural network be used to approximate sin(x) function?

Question

Since neural networks are universal function approximators, I am interested to know if a neural network with a simple activation function be used to approximate $\sin(x)$ Assuming the input is bounded.

Answer 1

With ReLU nodes, it is quite straight-forward to make triangular waves. For instance, if you have a layer of hidden nodes given by $$ \begin{align} a&: x\mapsto R(x)\\ b&: x\mapsto R(x-1)\\ c&: x\mapsto R(x-2) \end{align} $$ (where $R$ is the ReLU function) and then combine them like so in the output node: $$ a -2b + c $$ then this gives a function which is flat to the right of $0$, then goes straight up to the point $(1, 1)$, then straight down to $(2, 0)$, and from there it's flat. Which is to say, a single triangle with base along the $x$-axis of length $2$ and height $1$.

The main insight is that this triangle may be moved and scaled appropriately by changing the input linearly or scaling the output. For instance, $$ \begin{align} a&: x\mapsto R(x)\\ b&: x\mapsto R\left(x-\frac\pi2\right)\\ c&: x\mapsto R(x-\pi) \end{align} $$ and changing the output node to $$ \frac2\pi(a-2b+c) $$ gives you a triangle that aligns with the first half of a sine wave. Similarly, you can make a triangle for the second half of the sine wave, add them together, and get this result. Which is a good start.

So, if we want to do better, what can we do? Well, let's look at what's left. In other words, the difference between what we want and what we have. Here is a WolframAlpha plot of that. How can we fix this? More triangles!

This time we need four triangles. And at this stage we can choose our approach a little. Note that the waves here aren't completely symmetrical, so the maxima and minima aren't smack in the middle of the zeroes. Do we keep using symmetric triangles? That would make our expressions and our job a bit easier. If so, how tall do we make them? Or do we make triangles whose maximum coincides with the maximum of what we're after? Or maybe there is some other "best triangle fit" to look for.

I'll go for the symmetric triangles, with peak at the same height as the extrema. The first triangle is thus given by $$ \begin{align} d&: x\mapsto R(x)\\ e&: x\mapsto R\left(x-\frac\pi4\right)\\ f&: x\mapsto R\left(x-\frac\pi2\right) \end{align} $$ and to our output node we now add $$ 0.26803432(d - 2e + f) $$ to account for the first wave. And we can make similar triangles for the remaining waves. Unfortunately, WolframAlpha won't let me plot the result of all of them, so here is a plot using only the first of these four. You can see how we're already really closing in on the sine graph.

Speaking of choosing better-fitting triangles, though, maybe we should look for a better fit in the original two triangles as well? Perhaps using $0.8105695$ as a scaling constant in the output node is a truer approximation than $\frac2\pi$, as the two waves now have approximately the same area. Or maybe we want the maximal difference between the values to be as small as possible, rather than the areas as equal as possible? In that case, we want to use $0.724611$ as a scaling factor.