I am wondering why symmetric matrices are diagonalizable only by orthogonal matrices (and these orthogonal matrices by definition have orthonormal vectors). This is the proof but I don't really get the second part:
Why is this part true:
Finally, by Theorem 7.5, you can conclude that P^{-1}AP is diagonal. So, A is orthogonally diagonalizable.
How do we know it has n linearly independent eigenvectors?
On
You constructed $P$ to have columns being orthonormal eigenvectors. You should know a result stating that an set of orthogonal vectors being linearly independent. So the columns of $P$ are your $n$ linearly independent eigenvectors.
On
The identity matrix is symmetric, and is diagonalizable by any invertible matrix $P$ because $P^{1}IP=I$. So such a diagonalization is not necessarily unique.
If $A$ is symmetric, then it has an orthonormal basis $\{ d_1,d_2,\cdots,d_n \}$ of column eigenvectors with corresponding eigenvalues $\{ \lambda_1,\lambda_2,\cdots,\lambda_n \}$. In matrix notation
$$ A\left[\begin{array}{cccc}| & | & | & | & | \\ d_1 & d_2 & d_3 & \cdots & d_4 \\ | & | & | & | & | \end{array}\right] \\ = \left[\begin{array}{cccc} | & | & | & | & | \\ \lambda_1 d_1 & \lambda_2 d_2 & \lambda_3 d_3 & \cdots & \lambda_n d_n \\ | & | & | & | & | \end{array}\right] \\ = \left[\begin{array}{ccccc}| & | & | & | & | & \\ d_1 & d_2 & d_3 & \cdots & d_n \\ | & | & | & \vdots & |\end{array}\right] \left[\begin{array}{ccccc} \lambda_1 & 0 & 0 & \cdots & 0 \\ 0 & \lambda_2 & 0 & \cdots & 0 \\ 0 & 0 & \lambda_3 & \cdots & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & 0 & \cdots & \lambda_n \end{array}\right] $$ So $AU=UD$ or $A=UDU^{-1}$, where $D$ is diagonal. The matrix $U$ is orthogonal because the columns form an orthonormal basis, thereby forcing $U^{T}U=I$.
Conversely, if $A=UDU^{-1}$ where $D$ is diagonal and $U$ is an orthogonal matrix, then every column of $U$ is an eigenvector of $A$ because $AU=UD$.
In the proof of Thm 7.5, the columns vectors $p_i$ are the eigenvectors of $A$. (You may think about the matrix product on each side of $AP = PD$ in a column-wise manner.) Since $P^{-1}$ exists at the very beginning of the proof, $P$ is nonsingular. This forbids the column vectors $p_i$'s from being linearly dependent. (Otherwise, some column $p_j$ would be a linear combination of other columns, and this would give $\det(P) = 0$, contradicting the invertibility of $P$.)