Let
$ \qquad A = \begin{pmatrix}4 & 1 & 1 \\\ 1 & 2 & 3 \\\ 1 & 3 &2\end{pmatrix}$
Find $\smash{\displaystyle\max_{\mathbf{x}}} \frac{|(\mathbf{Ax}, \mathbf{x})|}{(\mathbf{x}, \mathbf{x})}$ , where $(·, ·)$ is a dot product of vectors and the maximization is performed over all $\mathbf{x} = \begin{bmatrix} x_1 & x_2 &x_3 \end{bmatrix}^T \in \mathbb{R}^3$, such that $\sum_{i=1}^{3}x_i = 0$.
How to find the max value? I tried taking random values of $x_i$ according to given condition and the dot product, $|(\mathbf{Ax}, \mathbf{x})|$ is always zero. Is the answer $0$ then?
$A$ has eigenvalues $6$, $3$, and $-1$ with corresponding eigenvectors $v_1=(1,1,1)/\sqrt{3}$, $v_2=(-2,1,1)/\sqrt{6}$, and $v_3=(0,-1,1)/\sqrt{2}$. Let $A = UDU^\top$ be the orthogonal eigendecomposition.
In particular, note that the condition $\sum_{i=1}^3 x_i = 0$ defines the span of the last two eigenvectors $v_2$ and $v_3$ (equivalently the orthogonal complement of the first eigenvector).
$$\max_{x : \sum_i x_i = 0} \frac{(Ax, x)}{(x,x)} = \max_{x : \sum_i x_i = 0} \frac{(D U^\top x, U^\top x)}{(x,x)} = \max_{y : y_1=0} \frac{(D y, y)}{(Uy, Uy)} = \max_{y : y_1=0} \frac{(D y, y)}{(y,y)} = \max_{y_2, y_3} \frac{3 y_2^2 - y_3^2}{y_2^2 + y_3^2} = 3.$$
The maximum is achieved by $y=(0,1,0)$, which corresponds to $x = v_2$