Let $A$ be an Hermitian matrix in a vector space $V$, and let $U\le V$ be a subspace of $V$.
If $U$ is invariant under $A$, then the maximum of $\langle x,Ax\rangle$ over all unit vectors $x\in U$ equals the largest eigenvalue of $A$ whose eigenvector is in $U$ (and we know $U$ is spanned by eigenvectors of $A$, as otherwise it wouldn't be invariant under its action). Similar ideas are used for example to prove the min-max principle.
What about subspaces $U$ that are not invariant under $A$? More precisely, is there a way to find $$\max_{x\in U}\frac{\langle x,Ax\rangle}{\|x\|^2}$$ for arbitrary subspaces $U$? Of course, feel free to remove the Hermitianity constraint if the problem is better posed in a more general setting.
Decomposing an arbitrary $x\in U$ as $x=\sum_k c_k x_k=\sum_j d_j u_j$ where $x_k$ are a basis of eigenvectors of $A$ and $u_k$ an orthonormal basis for $U$, we have $$\langle x,Ax\rangle = \sum_k \lambda_k |c_k|^2,$$ but the problem is that the maximisation is constrained to those coefficients $(c_k)$ such that $\sum_k c_k x_k\in U$,
Let P be the projector onto the subspace $U$. Then for all $x \in U$ we have $$ \begin{aligned} \langle x, A x \rangle &= \langle P x, A P x \rangle \\ &= \langle x, PAP x\rangle. \end{aligned} $$ Therefore, we have $$ \max_{x\in U}\frac{\langle x,Ax\rangle}{\|x\|^2} = \max_{x\in U}\frac{\langle x,PAPx\rangle}{\|x\|^2}. $$ Now $PAP$ is an operator that leaves the subspace $U$ invariant and so as noted in the question the maximum is given by the largest eigenvalue of $PAP$ whose eigenvector is in $U$. Note that we have an upper bound of $\lambda_{\max}(A)$ (largest eigenvalue of A) which can be seen be noticing the original problem is upper bounded by the same problem but with a maximization over the whole space $V$.