Hypothesise that $P$ is the symmetric matrix of some quadratic form $g(\mathbf{ x} ) = \mathbf{ x^TAx} $. Then $P$ is the orthogonal matrix consisting of orthogonal eigenvectors of $A$. Moreover, use this as the change of variable $\mathbf{ x = Py } \iff y = P^{-1}x = Px$. So $x : = (x, y)^T$ denotes the old axes and $y := (x', y')^T$ the new ones. Say $P = \begin{bmatrix} f_1 & \color{orangered}{s_1} \\ f_2 & \color{orangered}{s_2} \\ \end{bmatrix}$, where I use $f, s$ to denote the 1st and 2 colns of P.
$1.$ The Kolman and Hill book writes: "in term of the x- and y-axes, the x' lies along the first column of P, and the y' lies along the 2nd column of P".
Would someone please explain why? I concede that if I multiply out $y = Px$, then the result in equation form is : $x' = f_1x + f_2y$ and $y' = s_1x +s_2y$. But how to proceed?
$2.$ Lay P284 writes: "The order of the vectors in P is unimportant" because when constructing the D (= diagonal matrix) "from the corresponding eigenvalues," simply ensure that the eigenvalue in the ith column in D matches the eigenvector chosen by you for the ith column in P.
Wouldn't the order of P matter for sketching the quadratic form? If I reversed the columns of P, then the new x'-axis would now be the 2nd coln of P (in orange) and the y'-axis the 1st?
For 1. What you have put is somewhat incorrect, I.e. you should have $x=Py\Rightarrow y=P^{-1}x=P^Tx$, now let us denote $c_1,c_2$ as the columns of $P$ respectively. this gives us:
$x'=c_1^T(x,y)=f_1x+f_2y$ and $y'=c_2^T(x,y)=s_1x+s_2y$.
Which means that the new coordinate $x'$ is determined by the first column of $P$, and $y'$ by the second.
For 2. When we diagonalise $A$ we have $P^{-1}AP=D$ where the diagonal entries of $D$ are the eigenvalues of $A$ and the columns of $P$ are the corresponding orthogonal eigenvectors.
But this diagonalisation is not unique, since we can put the eigenvalues (the diagonal entries of $D$) in whichever order we like, but this means that the columns of $P$ would change.
So as long as the columns correspond to the correct eigenvalues, $P$ is well defined.
This does not affect the sketching, since $g(x)=x^TAx=x^TPDP^{-1}x=x^TPDP^Tx$, I.e the coordinates are changed, and then changed back again.