Assume $x$ are random vectors and $x'$ denotes the transpose of vector $x$. In Hansen's Econometrics an assumption of the Linear Predictor model is that ${\Bbb E} [xx']$ is positive definite.
I get that $xx'$ is always singular (as e.g. by multiplying the first row of the matrix by $x_2 \over x_1$, where $x' = (x_1, x_2, ...)$ it will equal the second row). I guess I cannot use the same trick with the Expectation sign on. As $E[x_1^2]*E[\frac{x_2}{x_1}] \neq E[x_1 x_2]$ (at least not necessarily). But this does not show yet that $E[xx']$ can be non-singular.
The argument I heard is that one can show $E[xx']$ to be positive semi-definite by showing
$$z'E[xx']z = E[z'xx'z] = E[(x'z)'(x'z)]=E[(x'z)^2]\ge 0$$, where $z \in \mathbb{R}^n$. And we are assuming it is in fact positive definite (i.e. no element of $x$, $x_i$ is a linear combination of another). Positive definite matrices are always non-singular. But why can I not simply do:$$z'xx'z = z'xx'z = (x'z)'(x'z)=(x'z)^2\ge 0$$ to show $xx'$ is also non-singular?
A positive semidefinite matrix is non-singular if and only if it is positive definite. If $E[xx']$ is positive definite, then for all $z \neq 0$, it holds that $z'E[xx']z > 0$, not merely that $z'E[xx']z \geq 0$ as you have written.
If you have a single vector $x \in \Bbb R^n$ (for $n > 1$), it is impossible for $(x'z)^2 > 0$ to hold for every vector $z \in \Bbb R^n$. In particular, we can always find a vector $z$ orthogonal to $x$, which is to say that $x'z = 0$, so that $(x'z)^2 = 0$. So, the matrix $xx'$ cannot be positive definite and so cannot be non-singular.
Circling back to $E[xx']$, the positive definiteness of this matrix is equivalent to the statement that $E[(x'z)^2]>0$ for every non-zero vector $z$. This in turn is equivalent to the statement that for every non-zero vector $z$, there is a non-zero probability of having $x'z \neq 0$. Since for a given $z$, the set of $x$ satisfying $x'z = 0$ is a hyperplane, what we're really saying is that there is no single hyperplane that captures the vector $x$ with probability $1$.
Within this framework, perhaps you can understand why $vv'$ (for a constant vector $v$) must be singular. You can think of $vv'$ as the matrix $E[xx']$ where $x = v$ with probability $1$ (or for a centered distribution, say $v$ or $-v$ each with probability $1/2$). Clearly, the set of possible values of $x$ fits within a single line, and so the set of possible values of $x$ can be contained within a single hyperplane.