Let the following non-Hermitian matrix $$A=\begin{bmatrix}1&2\\0&1\end{bmatrix}$$ I am asked to find its eigenvalues as well as the quantities $$a=\mathrm{max}\{\frac{\mathbf{x}^TA\mathbf{x}}{\mathbf{x}^T\mathbf{x}}:\ \mathbf{0}\neq\mathbf{x}\in\mathbb{R}^2\}$$ and $$b=\mathrm{max}\ \mathrm{Re}\{\frac{\mathbf{x}^*A\mathbf{x}}{\mathbf{x}^*\mathbf{x}}:\ \mathbf{0}\neq\mathbf{x}\in\mathbb{C}^2\}$$ and then somehow show that Rayleigh's Quotient Theorem is violated in this case. So, the characteristic polynomial is $p_A(t)=(t-1)^2$, so the eigenvalue is $\lambda=1$ with algebraic multiplicity of 2. Then, solving $A\mathbf{x}=\lambda\mathbf{x}$ for $\mathbf{x}=\begin{bmatrix}x_1&x_2\end{bmatrix}^T$ we see that $x_2=0$ and the eigenvectors are of the form $\mathbf{0}\neq\mathbf{x}=\begin{bmatrix}x_1&0\end{bmatrix}^T$, $x_1\in\mathbb{C}$.
For any real $\mathbf{x}\in\mathbb{R}^2$ $$\frac{\mathbf{x}^TA\mathbf{x}}{\mathbf{x}^T\mathbf{x}}=\dots=1$$ so $a=1$.
For any complex $\mathbf{x}\in\mathbb{C}^2$ $$\frac{\mathbf{x}^*A\mathbf{x}}{\mathbf{x}^*\mathbf{x}}=\dots=1$$ so $b=1$.
Rayleigh's Quotient Theorem (from Horn and Johnson's Matrix Analysis book) says
Let $A\in\mathrm{C}^{n\times n}$ be Hermitian, let the eigenvalues of $A$ be ordered as $\lambda_1\leq\dots\leq\lambda_n$, let $i_1,\dots,i_k$ be given integers with $1\leq i_1\leq\dots\leq i_k\leq n$, let $\mathbf{x}_{i_1},\dots,\mathbf{x}_{i_k}$ be orthonormal s.t. $A\mathbf{x}_{i_p}=\lambda_{i_p}\mathbf{x}_{i_p}$ for each $p=1,\dots,k$, and let $S=\mathrm{span}\{\mathbf{x}_{i_1},\dots,\mathbf{x}_{i_k}\}$. Then
(a)$$\lambda_{i_1}=\min\limits_{\{\mathbf{x}:\mathbf{0}\neq\mathbf{x}\in S\}}\frac{\mathbf{x}^*A\mathbf{x}}{\mathbf{x}^*\mathbf{x}}=\min\limits_{\{\mathbf{x}:\mathbf{0}\neq\mathbf{x}\in S\text{ and }||\mathbf{x}||_2=1\}}\mathbf{x}^*A\mathbf{x}\leq\max\limits_{\{\mathbf{x}:\mathbf{0}\neq\mathbf{x}\in S\text{ and }||\mathbf{x}||_2=1\}}\mathbf{x}^*A\mathbf{x}=\max\limits_{\{\mathbf{x}:\mathbf{0}\neq\mathbf{x}\in S\}}\frac{\mathbf{x}^*A\mathbf{x}}{\mathbf{x}^*\mathbf{x}}=\lambda_{i_k}$$
(b)$\lambda_{i_1}\leq\mathbf{x}^*A\mathbf{x}\leq\lambda_{i_k}$ for any unit vector $\mathbf{x}\in S$, with equality in the RHS (respectively, LHS) if and only if $A\mathbf{x}=\lambda_{i_k}\mathbf{x}$ (respectively, $A\mathbf{x}=\lambda_{i_1}\mathbf{x}$).
(c)$\lambda_\text{min}\leq\mathbf{x}^*A\mathbf{x}\leq\lambda_\text{max}$ for any unit vector $\mathbf{x}\in\mathbb{C}^n$, with equality in the RHS (respectively, LHS) if and only if $A\mathbf{x}=\lambda_\text{max}\mathbf{x}$ (respectively, $A\mathbf{x}=\lambda_\text{min}\mathbf{x}$). Moreover, $$\lambda_\text{max}=\max\limits_{\mathbf{x}\neq\mathbf{0}}\frac{\mathbf{x}^*A\mathbf{x}}{\mathbf{x}^*\mathbf{x}}\text{ and }\lambda_\text{min}=\min\limits_{\mathbf{x}\neq\mathbf{0}}\frac{\mathbf{x}^*A\mathbf{x}}{\mathbf{x}^*\mathbf{x}}$$
Now, I don't see which part of the theorem is violated in this case. I cannot find a suitable vector to show this. What do you suggest?
$$\begin{bmatrix}1&1\end{bmatrix} \begin{bmatrix}1&2\\0&1\end{bmatrix} \begin{bmatrix}1\\1\end{bmatrix} = 4 \neq 2 = \begin{bmatrix}1&1\end{bmatrix} \begin{bmatrix}1\\1\end{bmatrix}$$