During the last few weeks I've been having a light discussion with some peers at work about the applicability of the Strong Law of Large Numbers (SLLN) to a certain batch of data. Everybody mantains that "what we observe is due to the SLLN" but I, the only mathematician in the group, think that such theorem cannot safely be applied in this case. I will explain the situation and share with you my argument. I'd like to know your opinion on the matter, wether my argument is solid or has some fault and, ultimately, if it is correct to use the SLLN here.
The data we are studying is a performance metric of a prediction model. Such metric, let's call it $m$, can be thought as the ratio of accuracies between two sub-samplings of the data. In general, $m \in \left [ 0, \infty \right )$.
A certain plot depicting the value of $m$ for different sub-samples $S_1, \dots, S_n$ of the data shows that, the larger the sub-sample, the closer is its corresponding score, $m(S_k),\ k = 1, \dots, n$ to the score corresponding to the whole data sample, let's call it $m(S)$. This would appear to be a clear sign of the SLLN in action, however the trouble here is that each $S_k$ comes from a different source. To be precise on that, imagine we are taking blood samples from different people at different medical centers, then, for each $k$, $S_k$ comes from a different facility. Because of that, I argue that a "facility effect" might be mimicking the effect of the SLLN and fooling us all in our perceptions of the graph. Especially when the larger facilities are the ones with the closest values to $m(S)$.
Since so far it is just heuristics, let me expand on why I find it possible that the SLLN might not apply. We are working with a binary predictor, so basically we are trying to fit a function $y := f\left(x_1, \dots, x_l \right)$ with $y \in \left\{0,1 \right\}$. We may consider each instance $X:=\left(x_1, \dots, x_l \right) \in S$ to be a random vector, some of whose components are known to auto-correlate. That is, there is some index $r = 1, \dots, l$ such that a sequence of repetitions of $x_r$ is not an iid sequence of random variables. Because of this, a sequence of instances $\left\{ X_n \right\}_n \subseteq S$ is not an iid sequence of random vectors, and thus the SLLN cannot be safely applied to $\left\{ X_n \right\}_n$. Now, our performance metric $m$ can be thought as a function $g: \mathcal{P}(S) \longrightarrow \left [ 0, \infty \right )$; so for the whole sample $S$, $m(S) = g(S)$. My point here is that if the SLLN cannot be safely applied to $\left\{ X_n \right\}_n$ then it cannot be safely applied to $m$, unless $g$ has some outlandishly convenient properties that in any case shall be hard or impossible to check.
I haven't been doing serious math for a while now, so it might just be that I am mixing concepts and misusing them, so I'd like to know what are the communitiy's thoughts.