We have $X=UDV^T$ where $X\in R^{N\times p}$ and $U\in R^{N\times p}$ (we do SVD on data matrix X). After some computation, we get down to $X\hat{\beta}^{\text{least squares}}=...=UU^Ty$, which I can follow so far. But then the book says that $U^Ty$ are the coordinates of $y$ with respect to the orthonormal basis $U.$ I'm not following how $U^T$ and $U$ have the same orthonormal basis, since $U$ deals with a column space for $N$ dimensional vectors and $U^T$ deals with $p$ dimensional vectors, and was hoping somebody could help me clear up that confusion.
Is it that once we have $U^Ty$, this is the coordinate representation of $y$ if we had $U$ as the basis vectors instead of the standard basis vectors?
Let $u_1, \ldots, u_p$ be the columns of $U$. It is clear that these vector form a orthonormal basis of the column space of $U$ (or $A$). Since $y$ is an element of the column space of $A$, you can write it as $$y = \sum_{i=1}^p \alpha_i u_i.$$ Now how can you find the values of $\alpha_k$? Since the basis is orthonormal, $u_k^\intercal y = \sum_{i=1}^p \alpha_i u_k^\intercal u_i = \alpha_k$. Now if you want to see $\alpha_k$ as a vector you will have:
$$\alpha = U^\intercal y$$