I am exploring ideas and concepts in group theory, specifically in the context of linear algebra. Consider a full-rank square symmetric matrix $P \in \mathbb{R}^{n\times n}$ for which an eigendecomposition can be expressed as $$ P = Q\Lambda Q^T $$ where $Q$ is an orthogonal matrix and $\Lambda$ is a diagonal matrix. We know that such a decomposition is not unique. There can be several orthogonal matrices $Q$ that satisfy the above equation, arising from positive/negative choices for eigenvectors and permutations of these eigenvectors, leading to a total of $2^n + n!$ possible orthogonal matrices.
While I am aware that these matrices do not form a group in a strict mathematical sense due to the absence of an identity element, I am interested to find out if these matrices have any relevant connections or similarities to concepts in group theory. In particular, are there any meaningful links between this set of orthogonal matrices and certain finite subgroups of the orthogonal group $O(n)$?
Any insights on this or guidance towards relevant literature would be greatly appreciated.
You are correct in that even though these matrices don't form a group, they are closely related to one. Notice that if $Q$ and $R$ are two such matrices we have
$Q\Lambda Q^T = R\Lambda R^T$
and thus
$\Lambda = Q^T R \Lambda R^T Q$.
The set original matrices $M$ for which $\Lambda = M \Lambda M^T$ does form a group, and the set you described is a left coset of this group.
To give some more context: the mapping $X \mapsto QXQ^T$ defines a group action of $O(n)$ on the set of $n\times n$-matrices, and the subgroup of $O(n)$ for which the action fixes $\Lambda$ is called the isotropy group of $\Lambda$. An element for which $P = Q\Lambda Q^T$ for some $Q$ is said to be in the orbit of $\Lambda$. The elements in the orbit of $\Lambda$ are in bijection with the left cosets of the isotropy group.
PS: I believe you meant to write $2^n \cdot n!$. Also this group could be infinite if $P$ has an eigenvalue of multiplicity more than 1. For example if $P$ is the identity you get all of $O(n)$.