For example; can the dynamical systems theory help us find underlying general structures of the systems to make more accurate models?. Perhaps some data reflect these structures that are not evident by analyzing them from a purely statistical point of view.
Are there any works that relate to these areas? Or is it just a naive question? Thanks,
There are two obvious applications I can think of at the moment:
Directly Modelling/Controlling Temporal Data
Obviously, if the data itself can be modelled via a dynamical system, then the obvious thing to do is to blend learning into the time series model. E.g., using Gaussian processes or recurrent neural networks. This can be in different contexts: for instance, when you want to be able to find/discover patterns in dynamical data (e.g., [a]) or if you want to influence or control a dynamical system, you can for instance place an agent in a simulated environment and consider its ability to influence the system (e.g., [b]) within a deep RL framework.
[a]: Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems, Raissi et al
[b]: Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks, Depeweg et al
Model Learning Optimization as a Dynamical System
There is a large body of work within "deep learning" that treats the optimization itself as a dynamical system. We have some huge parameter vector $\Theta$ following a (stochastic) dynamical system, like $\partial_t \Theta_t = f(\Theta_t, \nabla_\Theta \mathcal{L}(X_t)), \;X_t\sim D_X$, where $f$ controls optimization aspects like momentum (or adaptive step sizing) and $X_t$ is randomly sampled data. Hence, this system is of great interest within ML (improvements in understanding it directly correspond to improving optimization, i.e. learning). I personally (:P) like work drawing parallels between these large-scale first-order optimization methods and physical dynamical system (like [c]). How to solve these non-linear dynamical systems numerically is a huge problem (e.g., [d]). One very fascinating (to me :3), more recent arena for this is studying the dynamics of GANs, which attempt to settle on a saddle point (Nash equilibrium) rather than a simple minimum, and thus have very interesting underlying dynamics (e.g., [e]).
[c]: The Physical Systems Behind Optimization Algorithms, Yang et al
[d]: Step Size Matters in Deep Learning, Nar & Sastry
[e]: GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, Heusel et al
Note: the references are just random ones to get you started since this area is enormous. You can use them to find earlier, more relevant (to you) literature, via the references within them. But let me know if you want more.