Take a Gauss map $G: [0,1] \longrightarrow [0,1]$ which is
$$G(x) = \frac{1}{x} \mod 1, 0<x<1$$ and $0$ if $x=0$. Let $\mu$ be the Gauss measure. For $x \in [0,1]$ let $[a_{1}(x), a_{2}(x),...]$ denote the continued fraction expansion of $x$. I would like to show that for any $\gamma>0$ we have for $\mu$-almost all $x \in [0,1]$ $$\lim_{n \longrightarrow \infty} \frac{a_{n}(x)}{n^{1+\gamma}} = 0.$$
Obviously, an application of Birkhoff's ergodic theorem is needed. I think we need to use the fact that for an ergodic measure preserving transformation $T$, with a measure $\mu$ and a function $f \in L^{1}$ we have $ \lim _{n \longrightarrow \infty} \frac{f(T^{n}(x)}{n} = 0$ for almost all $x \in \mu$. (This can be proved by considering partial sums of the form $\sum_{k=0}^{n-1} f(T^{k}(x)$).
But I don't know how to feel in the missing details - what function $f$ to choose, that this holds for all $\gamma >0$, etc. It looks like since we know $a_{n+1} = [\frac{1}{G^{n}(x)}]$ (where the square brackets denotes the integer part), we have to choose $f = \frac{[\frac{1}{x}]}{x^{\gamma}}$.
Is this right? I would appreciate any help and advice on how to make this rigorous.
Let $a(x) := \lfloor 1/x \rfloor$. As you noticed, we can't use Birkhoff theorem with the function $a$, as it is not integrable. We'll have to be more cunning. Since $a$ has a singularity in (essentially) $1/x$, note that $a^\alpha$ is integrable for all $\alpha \in [0,1)$. What does this gives us ?
First, note that for all $\alpha \in [0,1)$, the fonction $x \mapsto x^\alpha$ is concave and is $0$ at $0$. Hence, for all nonnegative $(a_0, \cdots, a_{n-1})$,
$$\sum_{k=0}^{n-1} (a_k^\alpha) \geq \left( \sum_{k=0}^{n-1} a_k \right)^\alpha,$$
or, if we put $1+\gamma/2 := 1/\alpha$, for all $\gamma >0$ :
$$\left( \frac{1}{n} \sum_{k=0}^{n-1} a_k^{\frac{1}{1+\gamma/2}} \right)^{1+\gamma/2} \geq \frac{1}{n^{1+\gamma/2}} \sum_{k=0}^{n-1} a_k \geq 0.$$
Let's apply this to $a_k := a \circ G^k$. We get:
$$\left( \frac{1}{n} \sum_{k=0}^{n-1} a^{\frac{1}{1+\gamma/2}} \circ G^k \right)^{1+\gamma/2} \geq \frac{1}{n^{1+\gamma/2}} \sum_{k=0}^{n-1} a \circ G^k \geq 0.$$
Thus, by Birkhoff's ergodic theorem, the sequence $\frac{1}{n^{1+\gamma/2}} \sum_{k=0}^{n-1} a \circ G^k$ is almost surely bounded. If we divide by $n^{\gamma/2}$, we get that, almost surely,
$$\lim_{n \to + \infty} \frac{1}{n^{1+\gamma}} \sum_{k=0}^{n-1} a \circ G^k = 0,$$
which is actually a stronger result than what you're looking for.