Is there a linear transformation that simultaneously reduces the pair of real quadratic forms $$x^2-y^2$$ and $$2xy$$ to diagonal forms?
My attempt I know that neither of these forms are positive definite, but if P=$$ \left[ \begin{array}{ c c } 2 & 0 \\ 1 & 0 \end{array} \right] $$ so that $x=2X$ and $y=X$ then both are diagonal. Is this correct?
First note that the given matrix $P$ is not invertible, so it cannot be used for diagonalization of a nondegenerate quadratic form.
The matrix representations of the two quadratic forms with respect to the standard basis are $$Q = \pmatrix{1&\\&-1}, \qquad Q' = \pmatrix{&1\\1&}.$$
Computing for a general change-of-basis matrix $P = \pmatrix{p_{ij}}$ the matrix representations $P^{\top} Q P, P^{\top} Q' P$ of the quadratic forms w.r.t. a general basis have respective off-diagonal components $$a = p_{11} p_{12} - p_{21} p_{22}, \qquad b = p_{11} p_{22} + p_{12} p_{21} ,$$ and by definition an invertible matrix $P$ simultaneously diagonalizes $Q, Q'$ iff $a = b = 0$.