My sample is a series of measurements of the variable $x$. Measurement t, $x_{t}$, is correlated with $x_{t-n}$. However, as n tends to infinite the correlation tends to zero.
If the sample grows, the mean value of $x$ tends to the expected value? What can I say about the variance of the population?
For simplicity assume that $E[X_t]=x$ for all $t$. The sample average of $n$ samples is $A_n = \frac{1}{n}\sum_{i=1}^n X_i$. Then $E[A_n]=x$ for all $n$, and the variance is $Var(A_n) = E[(A_n-x)^2] = E\left[\left(\frac{1}{n}\sum_{i=1}^n(X_i-x)\right)^2\right]$, which is: $$ \boxed{Var(A_n) = \frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n E[(X_i-x)(X_j-x)]} $$
1) Stationary covariance bound: If for all $i$ we have $|Cov(X_i, X_{i+k})| \leq h(k)$ for all $k \geq 0$ for some real-valued function $h(k)$ that satisfies $\lim_{k\rightarrow\infty} h(k)=0$, then the above variance formula gives: $$ Var(A_n) \leq \frac{1}{n^2}\sum_{i=1}^n\sum_{i=1}^nh(|i-j|)\rightarrow 0 $$ If $h(k)$ decays to 0 quickly enough, such as exponentially in $k$, so we can get $Var(A_n) \leq C/n^{\beta}$ for some constants $C>0$, $\beta>0$, and if the random variables $X_i$ are non-negative (or lower-bounded by some deterministic constant), then we can repeat the standard proof of the law of large numbers to prove that: $$ \lim_{n\rightarrow\infty} A_n = x \: \: \mbox{(with prob 1)}$$
2) Counter-example for non-stationary covariance bound: Suppose $x=0$ and samples $\{X_t\}_{t=1}^{\infty}$ are identically distributed with $Pr[X_t=1]=Pr[X_t=-1]=1/2$. However, the samples are generated over exponentially increasing frames. Samples in different frames are independent, but samples in the same frame are identical:
-Frame 1: Size $2$. Samples $t \in \{1, 2\}$. $X_1=X_2$.
-Frame 2: Size $4$. Samples $t \in \{3, 4, 5, 6\}$. $X_3=X_4=X_5=X_6$.
-Frame 3: Size $8$. Samples $t \in \{7, \ldots, 14\}$. $X_7=\ldots=X_{14}$
and so on, so Frame $k$ has size $2^k$.
Clearly for any $i$ we have $Cov(X_i,X_{i+n}) = 0$ for sufficiently large $n$ (since the samples are independent for sufficiently large $n$), and so indeed for any index $i$ we have $\lim_{n\rightarrow\infty} Cov(X_i,X_{i+n})=0$. However, the variance of $A_n$ does not tend to zero because after $k$ frames with $n(k) = \sum_{i=1}^k 2^i=2(2^k-1)$ samples we get: \begin{align} Var(A_{n(k)}) &= \frac{1}{n(k)^2}\sum_{i=1}^k(2^i)^2Var(X_1) \\ &= \frac{Var(X_1)(4/3)(4^k-1)}{n(k)^2} \\ &= \frac{Var(X_1)(4/3)(4^k-1)}{4(4^k - 2^{k+1}+1)} \end{align} Thus: $$ \lim_{k\rightarrow\infty} Var(A_{n(k)}) = \frac{Var(X_1)}{3} \neq 0$$