Suppose that $A$ and $B$ are real symmetric matrices and $A$ is invertible. Show that the matrix $A^{-1}B$ is diagonizable(similar to a diagonal matrix) if and only if there exists an invertible $P$ such that $P^{T}AP$ and $P^{T}BP$ are both diagonal matrices.
The 'if' part is straightforward; however the only if part seems quite hard. Can anyone help?
On
Thoughts so far on the "only if" direction:
Suppose that $A^{-1}B$ is diagonalizable. Select matrix $P_1$ such that $P_1^TAP_1$ is diagonal with $1$s, followed by $-1$s. We now have $$ (P_1^TAP_1)^{-1}(P_1^TBP_1) = P_1^{-1}(A^{-1}B)P_1. $$ Let $A_1 = P_1^TAP_1,B_1 = P_1^TBP_1, C_1 = P_1^{-1}(A^{-1}B)P_1$. We now have $A_1^{-1}B_1 = C_1$, so $B_1 = A_1C_1$ where $A_1$ has the form described, $B_1$ is symmetric, and $C_1 = A_1^{-1}B_1$ is diagonalizable.
I suspect that the trick from here is to select an orthogonal $U$ such that $U^TB_1U$ is diagonal and $UA_1 = A_1 U$. Note, however, that $UA_1 = A_1 U$ if and only if $U$ is block-diagonal with the same shape as $A_1 = \operatorname{diag}(I_p,-I_q)$. However, if $B_1$ is diagonalizable with such a $U$, then $B_1$ must also be block-diagonal of the same shape.
Long story short: in order to proceed with the proof in the way that I suspect it should go, we would need to show that if $A_1 = \operatorname{diag}(I_p,-I_q)$, $B_1$ is symmetric, and $A_1^{-1}B_1$ is diagonalizable, then $B_1$ must be block-diagonal (with block sizes $p$ and $q$). However, I don't see a clear reason that this should hold.
On
This is equivalent to part (a) of Theorem 4.5.17 of Horn and Johnson's Matrix Analysis, second edition.
Theorem: Suppose that $A$ and $B$ are Hermitian and $A$ is nonsingular. Let $C = A^{−1} B$. There is a nonsingular $S \in M_n$ and real diagonal matrices $\lambda,M$ such that $A = S\Lambda S^*$ and $B = SMS^*$ if and only if $C$ is diagonalizable and has real eigenvalues.
Your statement amounts to the case where $A$ and $B$ are also real matrices. The proof for the "only-if" direction is as follows:
Hint: Notice that $(P^T A P)^{-1} \cdot P^T B P = P^{-1} A^{-1} B P$ is the product of two diagonal matrices.