What is the orthogonal projection with expectation?

Question

I am reading an advanced econometrics textbook. When it talks about least squares, it says that the orthogonal projection of A onto Z is $P_Z(A)=Z^\prime E[ZZ^\prime]^{-1}E[ZA_k]$ and when A is a vector $A=(A_1,...,A_k)^\prime$, $P_Z(A)$ is defined as $$ P_Z(A)=(Z^\prime E[ZZ^\prime]^{-1}E[ZA_1],...,Z^\prime E[ZZ^\prime]^{-1}E[ZA_k])^\prime. $$ Although I know the basic projection matrix is $P=A(A^\prime A)^{-1}A^\prime$, I can't interpret the expectation in this formula. Could you help me?
And I think this is some basic statistic and matrix knowledge, so to pass the econometric course, could you give me some resources or advice to learn it?

Answer 1

Broadly speaking, in an inner product space, a projection of blah1 onto blah2 can be geometrically interpreted as an object living in blah2's subspace that is "closest" to blah1. How we measure "close" here would depend on the norm induced by our chosen inner product.

---

The projection in the context you describe is at the population level, choosing an inner product $<X,Y>\equiv \mathbb{E}[X,Y]$ for generic scalar square integrable random variables $X,Y$. The linear projection of $A\in\mathbb{R}$ onto $Z\in\mathbb{R}^p$ in this context is $Z'\beta$, where

$\beta=\arg\min_{\tilde \beta }\mathbb{E}[(A-Z'\tilde\beta)^2].$

You can show via calculus the solution to this minimization is

$\beta=(\mathbb{E}[ZZ'])^{-1}\mathbb{E}[ZA]$.

What you have written is an extension of this projection for $A\in \mathbb{R}^k.$

---

Let's return to the case $A$ is scalar. Another projection we can talk about is at sample level, choosing the usual Euclidean inner product between vectors. It is this projection that gives rise to the usual hat matrix you have seen in the context of least squares. To see this, suppose we have iid data $(A_i,Z_i),i=1,...,n$ and wish to use this data to best estimate the above population parameter $\beta$. Then we construct the data-aggregated objects ${\bf A}\equiv (A_1,...,A_n)'\in\mathbb{R}^n,{\bf Z}\equiv (Z_1,...,Z_n)'\in\mathbb{R}^{n\times p}$ and obtain our least squares estimator $\hat \beta$ via the linear projection of $\bf A$ onto $\bf Z$, given by ${\bf Z}\hat\beta$, where

$\hat \beta=\arg\min_{\tilde \beta }\|{\bf A}-{\bf Z}\tilde\beta \|^2.$

You can show via calculus the solution to this minimization is

$\hat \beta=(\bf Z'Z)^{-1}\bf Z'A$,

and thus $\bf Z(\bf Z'Z)^{-1}\bf Z'$ is called a hat (or projection) matrix since premultiplying this by $\bf A$ gives the projection of $\bf A$ onto $\bf Z$, or equivalently, fitted values for $A_i$ (putting a "hat" on them).

---

As an alternative to calculus, you can also solve the minimization problems by using an orthogonality condition (which follows from the projection theorem). For instance, in the case of the population level projection described above, it will be the case that $\forall \tilde\beta$, $Z'\tilde\beta$, or equivalently $\tilde\beta'Z$, is orthogonal to the optimal error:

$ 0=\mathbb{E}[\tilde \beta'Z(A-Z'\beta)]\quad \forall \tilde\beta\implies 0=\mathbb{E}[Z(A-Z'\beta)]\implies \beta=(\mathbb{E}[ZZ'])^{-1}\mathbb{E}[ZA]$.

---

You may find Hansen's resources helpful for further reading: https://www.ssc.wisc.edu/~bhansen/econometrics/