Would someone please explain "why $\det(P) = 1$ is required" and the general procedure of effecting this? Lay S7.2 didn't expound on this and neither does Kolman in S8.6-8.8.
Identify the graph of and write in standard form $9x^2 +y^2+ 6xy - 10\sqrt{10}x + 10\sqrt{10}y + 90 = 0$.
Rewrite this as $x^TAx + Bx + f = 0$ : $x^T \begin{bmatrix} 9 & 3 \\ 3 & 1 \\ \end{bmatrix} x + \begin{bmatrix} -10\sqrt{10} & -10\sqrt{10} \\ \end{bmatrix} x = 0.$
From symmetric A: $\lambda_1 = 0$ associates with $x_1 = (-1, 3)^T$. $\lambda_2 = 10$ associates with $v_2 = (3, 1)^T$.
I am not sure if this is what you are looking for, but as far as I can tell:
The question is asking you too re-write the quadratic expression in such a way that you can see what type of conic section it is and how it would be graphed.
The easiest way to do this is to write it as a rotated (ordinary) conic section. As far as the matrices go, orthogonal matrices $P$ have $\det P=\pm 1$. Those $R$ which are rotations preserve orientation, so they are the ones with $\det R=1$. Reflections have $\det R=-1$ and switch orientation, which means it would be a little harder to graph a conic from that type of representation.
You have the eigenvalues $(-1,3)^T$ and $(3,1)^T$ which, because we are dealing with quadratic forms (and hence symmetric matrices) are orthogonal as expected. Note that any scaled eigenvalue is still an eigenvalue with the same eigenvector (as long as you don't 'scale' by $0$). This is essentially a special case of the fact that eigenvectors form linear subspaces. Or algebraically, we have $$Av=\lambda v\implies A(cv)=c(Av)=c(\lambda v)=\lambda(cv)$$ so $c\mathbf{v}, c\in\mathbb{R}$ is still an eigenvector with respect to the eigenvalue $\lambda$.
So we have two tasks: to make a rotation matrix $R$, we have to (1) normalize the eigenbasis and (2) multiply various eigenvectors appropriately by the constant $-1$ so that the orthogonal matrix has $\det =1$ and is in fact a rotation.
After normalizing we get $$P=\begin{bmatrix}{-1\over \sqrt{10}} & {3\over \sqrt{10}} \\ {3\over \sqrt{10}} & {1\over \sqrt{10}} \end{bmatrix}$$ But this is a reflection. There are a few things we could do. We could switch the order of the eigenvalues so that the columns get switched. Or we could multiply either column by $-1$. That is how the book arrived at $$\begin{bmatrix}{1\over \sqrt{10}} & {3\over \sqrt{10}} \\ {-3\over \sqrt{10}} & {1\over \sqrt{10}} \end{bmatrix}$$ Note that multiplying rows, while changing the determinant, would alter the eigenvectors. But multiplying columns, as argued above, just produces new eigenvectors.
Now you complete the square to get (I believe) the equation for a parabola. You can graph the original equation by rotating this graph by the angle of $R$, which is in this case $\tan^{-1}{-3}\approx -72^\circ$.