$X$ a random variable; if $E[(b_i, X)^2] \lt \infty$ for basis $b_i$ of $(V, (\cdot, \cdot))$, then $\text{Cov}(X)$ exists.

Question

Let $V$ be a finite dimensional vectors space with inner product $(\cdot, \cdot)$, and $X$ a random variable in $V$. If $E[(b_i, X)^2] \lt \infty$ for all $i = 1\dots n$, then $\text{Cov}(X)$ exists.

The definition of $\text{Cov}(X)$ that I'm using is that if there exists a non-negative definite linear transformation $\Sigma : V \to V$ such that

$$ \text{cov}\{(x, X), (y,X)\} = (x, \Sigma y), \\\forall x,y \in V $$ then $\Sigma \equiv \text{Cov}(X)$.

Each $x$ (sim. $y$) may be composed as $x = \sum_i x_i b_i$. So that $(x,X) = \sum_i x_i (b_i, X)$ and $$ \text{cov}\{(x,X), (y,X)\} = \\ E[(x,X)(y,X)] - E[(x,X)]E[(y,X)] = \\ E[\sum_{i,j} x_i y_j (b_i, X)(b_j, X)] - (x, E[X])(y, E[Y]). $$

Where the last term is derived from the fact that $(x, X)$ has finite expectation when each $(b_i, X)^2$ does.

Not sure how to proceed or use more the fact that $E[(b_i, X)^2] \lt \infty$.

Answer 1

You have already noted that $|E[(b_i, X)]| < \infty$ due to $E[(b_i, X)^2] < \infty$ for each $i$.

With $x=\sum_i x_i b_i$ and $y = \sum_i y_i b_i$, you can check that $$\text{cov}\{(x,X), (y, X)\} = \text{cov}\left\{\sum_i x_i (b_i, X), \sum_j y_j (b_j, X)\right\} = \sum_i \sum_j x_i y_j \text{cov}\{(b_i, X), (b_j, X)\}$$ holds.

Then it remains to check that each covariance term is finite by using the moment conditions $E[(b_i, X)^2] < \infty$ and $|E[(b_i, X)]|<\infty$. The Cauchy-Schwarz inequality $E[(b_i, X)(b_j,X)] \le \sqrt{E[(b_i,X)^2] E[(b_j, X)^2]}$ may be helpful. Then, the desired matrix $\Sigma$ has entries $\Sigma_{ij} := \text{cov}\{(b_i, X), (b_j, X)\}$.

---

Regarding nonnegative-definiteness: you want to check that $(x,\Sigma x) \ge 0$ for any $x$. But by your work this is simply $\text{var}((x,X))$.

Answer 2

To expound on @angryavian 's answer and to show understanding.

First I need to check the linearity formula for covariance, that: $$ \text{cov}\{a X + bY, Z\} = a\cdot \text{cov}\{X, Z\} + b\cdot \text{cov}\{Y, Z\} $$

after which, by induction and symmetry of covariance we have bilinearity and @angryavian 's formula holds.

Proof.
$$ \text{cov}\{aX + bY, Z\} = E[(aX + bY)Z] - E[aX + bY]E[Z] \\ \text{(by linearity of } E) \ = aE[XZ] + bE[YZ] - aE[X]E[Z] - bE[Y]E[Z] \\ \text{(combining terms)} = a (E[XZ] - E[X]E[Z]) + b (E[YZ] - E[Y]E[Z]) \\ = a\cdot \text{cov}\{X, Z\} + b \cdot \text{cov}\{Y, Z\} $$

Next is how to derive the Cauchy-Schwarz formula involving expectation.

---

The missing link. According to that we, define a separate, seemingly unrelated inner product on the vector space $W$ of random variables in $V$ by: $$ [X, Y] := E[XY] $$

Then by Cauchy–Bunyakovsky–Schwarz inequality which applies to any inner product space, we have

$E[XY]^2 \leq E[X^2] E[Y^2]$.

---

The last part of @angryavian 's formula reads $\sum_{i,j} x_i y_j \text{cov}\{(b_i, X), (b_j, X)\} = (x, \Sigma y)$ for $\Sigma_{ij} = (\text{cov}\{(b_i, X), (b_j, X)\}$ since $(\Sigma y)_i = \sum_{j} \Sigma_{ij} y_j$ so that dotting by $x$ gives the formula equality.

This is true when each covariance term $\text{cov}\{(b_i, X), (b_j, X)\}$ is finite as @angryavian suggests.

But $\text{cov}\{(b_i, X), (b_j, Y)\} = E[(b_i,X)(b_j, X)] - E[(b_i, X)]E[(b_j, X)]$ which is finite since each $E[(b_i, X)]$ is, where of course you apply Caunch-Bunyakovsky-Schwarz to the term $E[(b_i, X) (b_j, X)]$.

---

Finally, $(x, \Sigma x) \geq 0$ or $\Sigma \equiv \text{Cov}(X)$ is non-negative definite ($\equiv$ positive semi-definite). We have:

$$ (x, \Sigma x) = \sum_{i,j} x_i x_j \text{cov}\{(b_i, X), (b_j, X)\} \\ = \sum_{i,j} x_i x_j \{ E[(b_i,X)(b_j,X)] - E[(b_i, X)]E[(b_j, X)] \} $$

... Difficulty arises!

---

Getting a hint from the text, I think we can rewrite the sum as: $$ \sum_{i} x_i (\Sigma x)_i = \sum_{i} x_i \{\sum_j \Sigma_{ij} x_j\} $$ (working on it). I'm trying to write as a sum of non-negative terms.