I am trying to get my head wrapped around this article in Wikipedia. The first definition given there is the covariance of a probability measure $\mathbf{P}$:
$$\mathrm{Cov}(x, y) = \int_{H} \langle x, z \rangle \langle y, z \rangle \, \mathrm{d} \mathbf{P} (z) \tag{$\ast$}$$
where $x, y \in H$ (a Hilbert space). I am used to the following definition of covariance:
$$\mathrm{Cov}(X, Y) = E((X - EX)(Y - EY))$$
where $X: \Omega \to \mathbb{R}$ and $Y: \Omega \to \mathbb{R}$ are two random variables defined on the corresponding probability space $(\Omega, \mathcal{F}, \mathbf{P})$, and
$$EX = \int_{\Omega} X(\omega) \, \mathrm{d} \mathbf{P} (\omega).$$
Question 1: Can anybody please translate my definition to the one in $(\ast)$? I would like to see it rewritten in the form of $(\ast)$, and I would really appreciate some discussion to get a better understanding of what is going on there.
At the end of the article, there is a definition of the covariance function of a random element $z$:
$$\mathrm{Cov}(x, y) = \int z(x) z(y) \, \mathrm{d} \mathbf{P} (z) = E(z(x) z(y)) \tag{$\ast\ast$}.$$
Question 2: First of all, why don't we subtract the expected values of $z$ at $x$ and $y$? Secondly, again, I do not really see a connection with my definition. Can anybody please give a clarification?
Thank you!
Regards, Ivan
The first definition is a special case of the second. Rather than a Hilbert space H, let's look at a Banach space $X$, and distinguish it from its dual space $X^*$. Every continuous linear functional $\varphi \in X^*$ is a random variable $\varphi : X \to \mathbb R$, so it makes sense to take expectations and covariances. We define the expectation of a functional by $\mathbb E[\varphi] = \int_X \varphi[x] \, \mathrm d \mathbf P(x)$, and the covariance of two functionals to be
$$\operatorname{cov}[\psi|\varphi] = \int_X \big( \psi[x] - \mathbb E[\psi]\big) \big( \varphi[x] - \mathbb E[\varphi]\big) \, \mathrm d \mathbf P(x).$$
Now, consider a probability measure $\mathbf P$ on a Hilbert space $X = H$. By the Riesz representation theorem, we know that the dual space $H^*$ is isomorphic to $H$, and all the functionals are of the form $\varphi_h[x] := \langle h, x \rangle$.
The mean can be represented by a single element $m \in H$, which is called the "Pettis integral" of $\mathbf P$. This element satisfies the property that $\mathbb E[\varphi_h] = \varphi_h[m] = \langle h, m \rangle$ for all $h \in H$.
Consequently,
$$\operatorname{cov}[\varphi_h|\varphi_{h'}] = \int_X \big\langle h, x - m \big\rangle \big\langle h', x-m \big\rangle \, \mathrm d \mathbf P(x).$$
This is the formula you were looking for. It's just a special case of the usual covariance formula, specialized to the setting where the random variables of interest are continuous linear functionals, and the probability space is a Hilbert space.