Crossposted here: https://stats.stackexchange.com/questions/432957/laws-of-large-numbers-for-sample-covariances-when-each-random-sample-are-just-kn
Let $X_1,...X_n$ be $p$-dimensional random sample, i.e. they're iid random vactors, each with mean $\mu$ and covariance matrix $\Sigma \in \mathbb{R}^{p \times p}$. Let $\bar{X_n}:= \frac{1}{n}(X_1+...+X_n)$ be their sample mean. Denote by $C_n$ the sample covariance matrix $C_n:= \frac{1}{n} \Sigma_{i=1}^{n}(X_i - \bar{X_n}) (X_i - \bar{X_n})'$, where ' denotes the transpose of a matrix.
Let's not assume anything for the moment, on the distribution of each $X_i$.
(1) Then, does $C_n \to \Sigma$ almost surely when $n \to \infty$? (General SLLN for sample covariance for any random samples).
(2) If we assume each $X_i$ to be normal, then is (1) true? (SLLN for covariance of normal random samples).
(3) If we assume each $X_i$ to be normal as well as $\mu = 0 \in \mathbb{R}^p, \Sigma= I_p$, then is (1) true? (SLLN for covariance of standard normal random samples )