Consider an integrable $n$ degree of freedom Hamiltonian $H$ with $n$ integrals $I_{1},...,I_{n}$ in involution. Take a level set $M$ of the integrals. According to Lioville-Arnold Theorem:
If on the level set $M$ the functions $dI_{i}$ are independent, then that level set is a smooth invariant manifold for $H$
If $M$ is compact and connected, it is diffeomorphic to an $n$-torus.
Action-angle variables may be introduced.
I don't understand what happens on "separatrix" level sets, and in particular which of the above conditions fail and why action-angle variables can't be used.
For example take a pendulum $H(q,p) = \frac{p^{2}}{2} + \cos q.$ Phase space is $\mathbb{T} \times \mathbb{R}$. Then the separatix level set is $H=1$, connecting the hyperbolic fixed point $(0,0)$ to itself (since we take $q$ modulo $2\pi$). Since we only have one integral $H = I$, what does this mean about "independence" requirement? And isn't $H=1$ a connected compact set?
Edit. I assume that condition 1) above is what fails in my pendulum example (since at the hyperbolic fixed point we $dH=(0,0)$ i.e. $dH$ vanishes; this point is connected by the separatrix hence by continuity on the separatrix level $dH$ does not satisfy condition 1).
Would be grateful if someone could shed more light on this.
In this case Liouville-Arnold doesn't tell much for the separitrix. Indeed as you've noticed $dH(0,0)=0$ and this means that $1$ is a critical value of $H$. Now, it is known that $H^{-1}(1)\setminus\{(0,0)\}$ is a submanifold of your phase space, but this is not compact, thus you can't apply the theorem. It is true that the preimage of a critical value can be a submanifold (and thus may be possible to apply the theorem) but in this case it doesn't happen.