I am trying to understand the linear algebra of PCA, and specifically what does it mean to be the $i$th principal component of some random vector $x$. I do know that this refers to the coordinate of $x$ in the new principal (the $i$th) direction. Right now my only confusion is regarding the why and how of coordinate projections of a vector. Let us see an example:
So if $x$ is a $p$-variate random vector with finite mean and $\Gamma$ is the eigenvector matrix of the covariance matrix of $x$ so that $\Sigma = \Gamma\Lambda\Gamma^T$, then allegedly the PCA tranformed version of $x$ is $y = \Gamma^T(x - \mathbb{E}(x))$. What is says is that each coordinate of $x$ in the new principal basis is given as the dot product between the normalized version of $x$ and the corresponding basis vector. Hence we could as well write $y = \sum_{i = 1}^{p}\left<x, \Gamma_i\right>\hat{\Gamma_i}$. And here is my dilemma: Since inner product tells you how similar two vectors are, it makes sense to form the new coordinates with the inner product and unit basis vector. On the other hand, since we have not assumed $\Gamma$ to be orthonormal, only orthogonal, this does not follow the text-book way of performing vector projection. Namely, shouldn't we take the inner product between the $i$th unit basis vector? Or is it so that you take the inner product with what ever the new basis vector is, unit or not, and then just multiply it with the corresponding unit direction vector?