Question related to Lagrange multipliers

Question

I am stuck with the following problem:

A is symmetric $n\times n$ matrix and $f(x)=(Ax)x$ for $x\in {\bf R}^n$. I need to show that the maximum and the minimum values of $f$ on the unit sphere ${x: |x|=1}$ are the largest and the smallest eigenvalues of $A$.

Here is how far I could go:

Let $e_1,e_2...e_n$ be a basis for A with eigenvalues $\lambda_1,\lambda_2,...\lambda_n$

Set $x=\sum a_je_j$ and $A=\sum \lambda_je_j$. Then, $x^2=\sum a_j^2e_j^2$ and $(Ax)x=\sum \lambda_ja_j^2$. If this is right, then I have reduced the problem to minimizing and maximizing $(Ax)x=\sum \lambda_ja_j^2$ with respect to $x^2=\sum a_j^2e_j^2=1$. I need to use to proceed, I guess, Lagrange multipliers, but I do not know how to do it. Could someone please check my ideas and help me to solve this problem finally?

Thanks in advance!

Answer 1

The currently accepted answer assumes the real spectral theorem, but this is not needed. In fact, Lagrange multiplies can be used to prove the real spectral theorem, and this is an essential step in the proof.

Let $f(x)=\langle Ax, x\rangle$.

Then

$ \begin{align*} f(x+v) &= \langle A(x+v),x+v \rangle\\ &= \langle Ax+Av,x+v \rangle\\ &=\langle Ax,x\rangle + \langle Av, x\rangle + \langle Ax,v \rangle+\langle Av,v \rangle\\ &=f(x)+2\langle Ax,v \rangle+\langle Av,v \rangle \text{ b/c $A$ is symmetric } \end{align*} $

This shows that $\nabla f = 2Ax$, since $\langle Av,v\rangle$ is of order $|v|^2$. (In fact $|\langle Av,v \rangle| \leq |A|_{op}|v|^2$ where $|A|_{op}$ is the "operator norm" of $A$).

We are trying to maximize $f$ with respect to the constraint that $g(x) = |x|^2$ is equal to $1$.

We know that $f$ achieves a maximum at some point $\nu$ on this unit sphere, since the unit sphere is compact. Lagrange multipliers then tell us that this maximum satisfies the property

$$ \nabla f(\nu) = \lambda \nabla g(\nu) $$

But $\nabla g(\nu) = 2\nu$

So $2A\nu = 2\lambda\nu$. Dividing by $2$

$$ A\nu = \lambda \nu $$

Which exactly states that $\nu$ is an eigenvector.

Now to prove the real spectral theorem, you could show that $A$ restricts to a symmetric operator on the perpendicular space to $\nu$. Then an induction on the dimension of the space gives the real spectral theorem.

---

I guess you actually ask for something a little different. But this argument shows that the maximum value must occur at an eigenvector. Moreover, the value of $f$ at an eigenvector of modulus $1$ is just the eigenvalue. So the maximum occurs at the eigenvector with the greatest eigenvalue.

Answer 2

$n \times n$ (real) symmetric matrices have the property that the eigenvectors are orthogonal and there are $n$ of them. So if the $e_j$ are an orthonormal basis of eigenvectors, then indeed we can write any vector $x \in \mathbb{R}^n$ as $x = \sum a_n e_n$ and thus what I take it you mean the inner product $f(x) = \langle Ax, x \rangle = \sum \lambda_n a_n^2$, subject to $g(x) = \|x\|^2 = \sum a_n^2 = 1$.

Applying Lagrangian method, we want to find a Lagrange multiplier $\mu$ such that $\nabla f = -\mu \nabla g$ satisfying the constraint $g(x) = 1$, i.e., $(2\lambda_1 a_1, 2\lambda_2 a_2, \cdots, 2\lambda_n a_n) = -\mu(2 a_1, 2 a_2, \cdots, 2 a_n)$ and $\sum a_n^2 = 1$.

If the eigenvalues $\lambda_j$ are all distinct, then there will only exist values of $\mu$ when $a_j = \pm 1$ for each $j$ in turn and $a_k = 0$ for $k \neq j$. In other words, all maxima and minima must occur along a basis vector, the value of any such local max or min being $f(x) = \lambda_j.(\pm 1)^2 = \lambda_j$. It then follows that the global maximum is $\max_j{\lambda_j}$ and the global minimum is $\min_j{\lambda_j}$.

The one thing you now need to finesse is what happens if the eigenvalues are not all distinct, i.e., if there are eigenspaces of dimension greater than one.