How to show E(XX') is not invertible

Question

Supposed that

$$ \boldsymbol x= \begin{bmatrix} 1 \\ \alpha_1+\alpha_2x_3 \\ x_3 \end{bmatrix} $$

where $x_2=\alpha_1+\alpha_2x_3$ Show that $E(\boldsymbol{xx^{'}})$ is not invertible.

Here is my partible solution:

$$ \boldsymbol {xx'}= \begin{bmatrix} 1 \\ \alpha_1+\alpha_2x_3 \\ x_3 \end{bmatrix} \begin{bmatrix} 1 & \alpha_1+\alpha_2x_3 & x_3 \end{bmatrix}= \begin{bmatrix} 1 & \alpha_1+\alpha_2x_3 & x_3 \\ \alpha_1+\alpha_2x_3 & (\alpha_1+\alpha_2x_3)^2 & (\alpha_1+\alpha_2x_3)x_3 \\ x_3 & (\alpha_1+\alpha_2x_3)x_3 & x^2_3 \end{bmatrix} $$

Is it enough to say that we can obtain the third row by multipling $x_3$ to the first row?

Answer 1

Assume in your example, $x_{3}$ is random, while $\alpha_{1},\alpha_{2}$ are not.

You have shown $xx^{\top}$ is always singular. However, if your $E$ here means expectation, then it doesn't imply that $\mathbb{E}[xx^{\top}]$ is also singular.

In fact, you can prove the following lemma:

Let $x$ be a $k$-dimensional random vector, then $\mathbb{E}[xx^{\top}]$ is singular if and only if there exists a non-random vector $a\in\mathbb{R}^{k}\setminus \{0\}$ such that $a^{\top}x=0$ almost surely.

Proof. The "if" part: Since $\mathbb{E}[xx^{\top}]$ is always positive semidefinite, to prove it is singular, it suffices to show it is not positive definite. Suppose $\exists a\in\mathbb{R}^{k}\setminus\{0\}$ such that $a^{\top}x=0$ alomost surely, then we have $a^{\top}\mathbb{E}[xx^{\top}]a=\mathbb{E}[a^{\top}xx^{\top}x]=\mathrm{Var}(a^{\top}x)=0$, which means $\mathbb{E}[xx^{\top}]$ is not positive definite.

The "only if" part: Suppose $\mathbb{E}[xx^{\top}]$ is singular, then its columns doesn't have full rank. Therefore $\exists a\in\mathbb{R}^{k}\setminus\{0\}$ such that $\mathbb{E}[xx^{\top}]a=0$, thus $a^{\top}\mathbb{E}[xx^{\top}]a=0$, which implies $\mathbb{E}\bigl[|a^{\top}x|^{2}\bigr]=\mathbb{E}[a^{\top}xx^{\top}a]=a^{\top}\mathbb{E}[xx^{\top}]a=0$. Note that $|a^{\top}x|^{2}\geq0$, therefore we must have $a^{\top}x=0$ almost surely. This completes the proof.

Recall in your case $x=(1,\alpha_{1}+\alpha_{2}x_{3},x_{3})^{\top}$, by taking $a=(\alpha_{1},-1,\alpha_{2})^{\top}$, apply above lemma, you can prove $\mathbb{E}[xx^{\top}]$ is singular.

Answer 2

You are correct in your logic. Here is how you can show it mathematically:

If we multiply one row of a matrix $A$ by a nonzero scalar $x_3$, then the determinant of the resulting matrix $B$ is the determinant of the original matrix multiplied by this scalar. The determinant of the resulting matrix $B$ will be zero (because it has two identical rows). Therefore, the determinant of the original matrix $A$ must also have been zero. Hence, it is singular.