
I know that irrational number can be approximated by p/q and error less than 1/q^2. But I still cannot give a rigorous proof to this problem.
And how to show that the difference between the left and the right is bounded by C*N^(-0.5), where C is a constant determined by f and ω?
Try to find information about irrational rotation (in the circle) and Birkhoff's ergodic theorem. That should help understanding (and proving) this convergence. Also feel free to look at a book on ergodic theory like Walters, Introduction to Ergodic Theory or the more modern book by Einsiedler and Ward...