There is some important background to go over before this question will make any kind of sense, so before calling me out on my poor understanding of dynamical systems, chaos, and differentiabilty, please take the time to read the whole question, and feel free to offer corrections or improvements if you see something that doesn't quite make sense.
Or just skip to the end. I'm not the boss of you.
About a week ago, I posed a simple question to myself: what is chaos, and can it be quantified? Specifically, I was curious as to why certain "well-behaved" systems are not generally considered "chaotic" despite considerable variation in the values of there solutions at a sufficient "distance" from the initial condition.
I spent several days reading introductory material, coming up with various definitions, and testing them against well known chaotic-systems. After several attempts at formalizing the notion of "sensitivity to initial conditions", I eventually settled on the following:
For a system $S$, let $\text{Sol}(S,\mathbf{x}_0)$ be the solution to $S$ with initial condition $\mathbf{x}_0$ (assuming the solution exists and is unique).
A system $T$ is orderly iff for a given initial condition $\mathbf{c}$, for real numbers $t,\varepsilon>0$, there exists a real number $\delta>0$ such that for $0<t'<t$, the difference between $\text{Sol}(T,\mathbf{c})(t')$ and $\text{Sol}(T,\mathbf{c}')(t')$ is less than $\varepsilon$ whenever the difference in between $\mathbf{c}$ and $\mathbf{c'}$ is less than $\delta$.
Symbolically, $T$ is orderly iff:
$$\forall \mathbf{c}.\forall t.\forall\varepsilon>0.\exists \delta> 0.\forall\mathbf{c}'.\forall t'<t.\Vert \mathbf{c}'-\mathbf{c}\Vert<\delta\implies\Vert\text{Sol}(T,\mathbf{c}')(t')-\text{Sol}(T,\mathbf{c})(t')\Vert<\varepsilon$$
A system is chaotic iff it is not orderly.
While I was not thinking about it at the time (the use of $\varepsilon$ and $\delta$ was out of habit rather than intent) I realise that this corresponds almost exactly to the statement "$\text{Sol}$ is ($\varepsilon$,$\delta$)-continuous over the set of initial conditions." This motivated me to try and come up with a more analytical definition of "system" - one that would allow concepts like "continuity over the set of initial conditions" to be expressed more precisely:
A system is a function $S:I\to Y^X$, where $I$ is the space of initial conditions and $Y^X$ is the set of functions from $X\to Y$.
A parametrization of a system $S$ is a set of expressions uniquely satisfied by $S$. For example, $\{f'(x)=f(x),f(0)=c\}$ is a parametrization of the system $S:\Bbb{R}\to\Bbb{R}^\Bbb{R}; S(c)=\lambda x.ce^x$.
From this point of view, a system $S:I\to Y^X$ is chaotic iff it is not continuous (different definitions of "continous" can be implemented analogously to those used in analysis with minimal effort).
Furthermore, $S$ is chaotic locally over $E\subset I$ iff $S$ is not continuous over $E$.
Now, it seems to me that this definition is much narrower than I originally intended. While I have not been able to prove as such (for want of analytical solutions), I have reason to believe that at least one "chaotic system," the Lorenz system, is not chaotic under this definition.
A cursory examination of numerical solutions for initial condition $c=(1,1,1)$ and $\Delta c = 10^{-n}(1,1,1)$ shows that as the difference between $c$ and $\Delta c$ approaches zero, the difference between $S(c)$ and $S(\Delta c)$ approaches the constant function $\lambda x. 0$. This is shown in the graphic below (note that these were made prior to coming up with the present definition of "system").
[Plot of the difference between solutions to the Lorenz system as $\Delta c\to0$]
[Solutions become pathological beyond $t\approx 24$, but this may be a computational artifact. I would be curious to see if anyone knows of a way to show that there is some distinct value of $t$ beyond which all solutions diverge regardless of $\Delta c$]
This would suggest that the Lorenz system is not chaotic. In fact it would seem that many, perhaps even most, classically "chaotic" systems are not truly chaotic by this definition.
Should we declare then that chaos theory is more-or-less bunk, then?
Well I don't think so. Instead of taking chaos to be a property which divides the universe into the orderly and disorderly, it makes intuitive sense to me that "chaos" fits into some kind of scale. It is not difficult to imagine a hierarchy of systems proceeding from least to most chaotic. As a starting point, we might take:
continuously differentiable $\subset\cdots\subset$ $n$-times differentiable $\subset\cdots\subset$ differentiable $\subset$ continuous $\subset\cdots\subset$ unrestricted
and consider the complement of the preceding classes in each succeeding class is "more" chaotic than the previous (that is, the most chaotic system is non-differentiable, everywhere discontinuous, etc.).
Finally, the question:
If I had the time, I would like to take a broad survey of the systems most commonly described as chaotic; then, one-by-one I could assign each system to an "order" class, adding criteria as I go until I had a definitive order-chaos scale.
Unfortunately, I am not being paid to research chaos theory, and the extent of my interaction with even well-behaved systems of differential equations can be summarised as "Oh! I've heard about those!"
So I'll settle for this:
While it is relatively easy - for a sufficiently inclusive "relative" - to show that certain chaotic systems are "continuous" (in the sense described above), it is another matter entirely to show that the same system is differentiable.
Based on the limited evidence available, I suspect that the most commonly studied chaotic systems (Lorenz system, Rossler system, n-body system, etc.) are continuous, but I do not see anything that would strongly indicate whether or not they are differentiable (the apparent smoothness of the difference in solutions may be due to limited precision).
Is it the case that the "chaotic systems" commonly considered in chaos theory are non-differentiable?
If not, then what would be a reasonable criteria for the "least" chaotic class of systems in the hierarchy?
Note:
I would guess that "if $\mathbf{f}$ is smooth and $\mathbf{g}$, then $\Vert\mathbf{f}-\mathbf{g}\Vert$ is smooth" - and this does suggest that many chaotic systems are differentiable. However, I'm not sure how well this works in the limiting case, especially considering the possibility of continuous, non-differentiable regions (the pathological portion of the Lorenz system might be one such region).