Covariance of a Vector-Valued Random Variable

Question

I'm reading through Andrew Ng Lecture Notes for CS229 and he makes the statement that, for a random variable $Z \in \mathbb{R}^{n}$,

\begin{align} Cov(Z) &= E[(Z - E[Z])(Z - E[Z])^{T}]\\ &= E[ZZ^{T}] - E[Z](E[Z])^{T} \end{align}

I've started with: \begin{align} Cov(Z) &= E[(Z - E[Z])(Z - E[Z])^{T}]\\ &= E[(ZZ^{T}) - 2(E[Z]Z^T) + E[Z](E[Z])^{T}] \end{align}

But I'm not sure how to continue.

Answer 1

Just derive as an extension of the single variable case. Derive Variance(X), where X is a random variable, as a function of E(X), you get E($X^2$) - $(E(X))^2$. Just extend to the multivariate case, while remembering some of the other hints listed in above comments.

Answer 2

You have (with the middle step added)

$\qquad\begin{align} \mathsf{Cov}(Z) &= \mathsf E[(Z - \mathsf E[Z])(Z - \mathsf E[Z])^\top] \\ &= \mathsf E\big[ZZ^\top - Z\,\mathsf E[Z]^\top- \mathsf E[Z]\,Z^\top + \mathsf E[Z]\,\mathsf E[Z]^\top\big] \\ &= \mathsf E\big[ZZ^\top - 2\,\mathsf E[Z]\,Z^\top + \mathsf E[Z]\,\mathsf E[Z]^\top\big] &&\text{because }Z\,\mathsf E[Z]^\top=\mathsf E[Z]\,Z^\top \end{align}$

But for the same reason:

$\qquad\begin{align}\mathsf{Cov}(Z)&= \mathsf E\big[ZZ^\top - 2\,Z\,\mathsf E[Z]^\top + \mathsf E[Z]\,\mathsf E[Z]^\top\big] \end{align}$

Then it is just a matter of using Linearity of Expectation

$\qquad\begin{align}\mathsf{Cov}(Z)&= \mathsf E[ZZ^\top] - 2\,\mathsf E[Z]\,\mathsf E[Z]^\top + \mathsf E[Z]\,\mathsf E[Z]^\top\\&=\mathsf E[ZZ^\top] - \mathsf E[Z]\,\mathsf E[Z]^\top \end{align}$