Claim about positive definiteness of $X'X$ and the rank of $X$

Question

In an econometrics note it says:

Suppose that the rank of $X \in \mathbb R^{N \times K}$ is $K$, so that $X'X$ is positive definite.

I am trying to convince myself that this is true. I know that a p.d. matrix is invertible. Moreover, if rank of $X$ is $k$ then $X'X$ is full rank and invertible. However, doesn't any arbitrary matrix $M = X'X$ positive definite by definition? Because for any vector $y$ we have $y^\top M y = y^\top X' X y = (Xy)^\top Xy \geq 0$.

So I am not sure if the claim is wrong or if I am confused.

Answer 1

The rank condition on $X$ implies $N\ge K $, which makes sense in the context of the econometric linear regression origin of the question, even if not explicitly stated by the OP.

Let $M=X'X$. Without the rank condition on $X$ the matrix $M$ is positive semi-definite. With the rank condition, $M$ is also positive definite. It should be clear that $a'Ma=a'(X'X)a=(Xa)'(Xa)=\|Xa\|^2\ge0$; what you need to show is that $a'Ma=0$ implies $a=0$. But if $a'Ma=0$ then $\|Xa\|^2=0$ and so $Xa=0$. By the rank $K$ condition on $X$, we know $a=0$. And, under the rank condition, $M$ is invertible.

Answer 2

tl;dr: Positive semi-definite, yes. Positive definite, no.

You are right, every matrix $M=X'X$ is positive semi-definite. However, the claim here is positive definite, which is stronger:

• Positive semi-definite: for all $y$, $y^T M y\geq 0.$
• Positive definite: for all $y$, $y^T M y > 0.$

See. e.g., the Wikipedia article.