For many applications in applied mathematics (e.g. Machine Learning), we tend to optimize functions of interest using the Gradient Descent Algorithm. As we know, the Gradient Descent Algorithm takes its roots from Taylor Series : Gradient Descent creates a first order Taylor Expansion of the function of interest, and determines the next direction to search for this functions minimum based on Tylor Expansion. This approach is typically taken when dealing with optimizing the Loss Functions for Neural Networks.
As we also know, there are different methods in mathematics which can be used to approximate functions. For instance, instead of the Taylor Approximation, we can also use the Fourier Approximation instead. This has always lead me to wonder: Why the optimization algorithms used for Neural Networks based on approximations from Taylor Series instead of Fourier Approximations.
Thinking out loud, a few ideas come to mind:
- Fourier Approximations are typically intended for sinusoidal and periodic functions
- Taylor Series are said to approximate a function pointwise, whereas Fourier Approximations approximate a function globally (https://sites.oxy.edu/ron/math/120/03/labs/lab8.PDF)
However, this then brings me to the following question:
Are we able to compare (e.g. "lower bound") to pointwise error of Fourier Approximations to that of Taylor Series?
Are we able to compare (e.g. "lower bound") to global error of Fourier Approximations to that of Taylor Series?
Based on the amount of literature and research being done that highlights the success of Optimization Algorithms using Taylor Approximation (e.g. Gradient Descent) in Machine Learning - I am suspecting that Fourier Approximations are likely not suitable in this regard.
Thanks!