Let $U:H \to H$ be an unitary operator on a Hilbert space $H$. Suppose that $x \in H$ is orthogonal to all the eigenvectors of $U$. I'd like to prove that
$$ \lim_{N \to \infty} \frac{1}{N} \sum_{n=1}^N \lvert (U^n x, x) \rvert = 0. $$
So far I have no idea how to start. Could someone point me in the right direction?
By the spectral theorem for unitary operators $U=\int_0^{2\pi}e^{it}dE_t$, where $E_t$ is a right continuous monotone projector valued function, and by the functional calculus $U^n=\int_0^{2\pi}e^{int}dE_t$. The jump $E_t-E_{t-0}$ at $t$ is the orthoprojector onto the $e^{it}$ eigenspace, and $(E_t-E_{t-0})u=0$ if $u$ is orthogonal to that eigenspace. Since $x$ is orthogonal to all eigenspaces $F(t):=(E_t\,x,x)$ is a continuous distribution function, and $c_n:=(U^nx,x)=\int_0^{2\pi}e^{int}dF(t)$ are its Fourier-Stieltjes coefficients.
By a theorem of Wiener $\frac1N\sum_{n=1}^N|c_n|^2\xrightarrow[N\to\infty]{}0$, and this is stronger than your claim because by the power means inequality $\frac1N\sum_{n=1}^N|(U^nx,x)|=\frac1N\sum_{n=1}^N|c_n|\leq\Big(\frac1N\sum_{n=1}^N|c_n|^2\Big)^{\frac12}\xrightarrow[N\to\infty]{}0$.
Wiener's theorem is on the first page of the paper I linked, it can be viewed by clicking on Look Inside button. The paper references Y. Katznelson, Introduction to Harmonic Analysis, John Wiley and Sons, 1968, p.42.