Let $H(I_1,I_2,\varphi_1,\varphi_2,t) = H_0 + H_1$ where $I_j$ is the conjugate variable of $\varphi_j$ for $j =1 ,2$ and $H_0 =\frac{1}{2} I_1^2 + \varepsilon (\cos\varphi_1 - 1)$ and $H_1 = \frac{1}{2} I_2^2$.
I am trying to find the stable/unstable manifold of this system. The paper I am reading claims it is the three dimensional manifold described by $H_0 = 0$ , $H_1 = \frac{1}{2}\omega^2$ for some irrational $\omega$. I simply can't see how he got this. I see that it is 3 dimensional as $\varphi_2,t$ can vary without affecting H.
If it helps: notice that $H_0$ is hamiltonian of the pendulum:
$$\dot{I_1} = - \varepsilon \sin\varphi_1 \text{ and } \dot{\varphi_1} = I_1$$ and so $\
Just to give a bit of background:
I am trying to fill in the details that Arnold left out in his 1964 paper "Instability of dynamical systems with several degrees of freedom". I can't seem to find any articles or references that aids with the rigorously checking out the details. So also if anyone could recommend anything to help that they have come across.
Since you identified the motion of $H_1$ I assume you understand the two equations you've given. The third is $t = -H$. Three integrals, three degrees of freedom, so this system is integrable, and stable, in the terminology of J. Moser. It's a model system that Arnold sets up in order to perturb it and show the behavior of a near integrable system nearby. I'm going to assume that this third integral is the one you're not seeing that shows the system lies for each $\epsilon$ on a 2-torus in the 3-torus of the whole system, like you have said, for most values it wraps around that torus densely (irrational $\omega$). How he got this, if you're referring to Arnold, is that he created it that way as a starting point, using the pendulum you correctly pointed out. It is a model system.
What he's going to do next is perturb it. The full Hamiltonian is $$ H=1/2(I_1^2 + I_2^2) + \epsilon (\cos\phi_1 - 1)(1 + \mu\sin\phi_2 + \mu\cos t). $$
As $\mu$ moves away from zero, the regular orbits break creating rings of stable and unstable periodic points, starting with those with the most badly spaced continued fraction expansions, and going to the least (which is the golden mean irrational orbit). The same homoclinic tangle as in the KAM theorem results, with one major difference from a 2-degree of freedom system: The stable orbits no longer wall them off from each other (there's no Jordan curve theorem in 3 dimensions, essentially). That is the point of Arnold's paper.
I think it would help if you looked at his papers and those of his colleagues in chronological order and remember what is and isn't therefore known when he does the paper, especially since Arnold's papers, as opposed to his books, are not easy to get through because they weren't reviewed for clarity only correctness.
When he writes this paper, the first version of the KAM theorem is done, the nature of the homoclinic tangle is still a conjecture, albeit one going back to Poincare's drawings. What comes next is Moser's twist theorem, and Emil Zehnder's proof that the orbits do indeed tangle and form homoclinic and heteroclinic orbits, and Smale's proof of what that looks like (the Beaches of Brazil paper), and then finally a Holmes and Marsden showing that Arnold diffusion actually occurs in a specific system.
Hope this helps. Some sources you might look at are J. Moser, Stable and Random Motions in Dynamical Systems, Lichtenberg and Lieberman, Regular and Stochastic Motion, and Abraham and Marsden, Foundation of Mechanics. The first is a very clear exposition of a lot of the underlying theory for these KAM systems, the second has a discussion of this equation in Chapter 6, and the third has beautifully rendered pictures of what the system you are working looks like on pp. 584-5. Also, in Arnold's Mathematical Methods in Classical Mechanics, he explains the winding around tori in detail at section 51, Chapter 10.