Suppose that
$$ \left( \begin{matrix} 1 & 0 & 1 \\ 1 & 1 & 0 \\ 0 & 1 & -1 \\ \end{matrix}\right) $$
calculate the maximum and minimum value of ||Ax|| on the sphere {x ∈ $R^3$: ||x|| = 1}.
I have $A^TA$ is symmetric then use it to get the max and min.But how to show the norm of $\mathbf Ax$
On
The minimum is $0$; just take $x=\frac1{\sqrt3}(-1,1,1)$.
The maximum is $\sqrt3$. This can be determined applying the method of Lagrange multipliers. If $x=(a,b,c)$ is such that $\|x\|=1$, then $a^2+b^2+c^2=1$. On the other hand, $\|Ax\|^2=2+2(ab+ac-bc)$. Define $f(a,b,c)=2+2(ab+ac-bc)$ and define $g(a,b,c)=a^2+b^2+c^2$. Applying the method of Lagrange multipliers in order to find the maximum and the minimum of $f$, you are led to system of equations
$\displaystyle\left\{\begin{array}{l}b+c=2\lambda a\\a-c=2\lambda b\\ a-b=2\lambda c\\a^2+b^2+c^2=1\end{array}\right.$
This system has $8$ solutions. For $6$ of them, the value of $f$ at that point is $3$ and therefore the norm of $A$ times that point is $\sqrt3$. One such point is $\left(\frac{1}{6} \left(-3-\sqrt{3}\right),-\frac{1}{\sqrt{3}},\frac {1}{6} \left(-3+\sqrt{3}\right)\right)$. At the remaining $2$ points, the value of $f$ is $0$.
Another method that I find easier (but specialized to this situation only) is using the eigenvalue method. You want to find the maximum and minimum of $||Ax||$. Note that this is the same as optimizing $||Ax||^2 = (Ax)\cdot (Ax) = (Ax)^T(Ax) = x^T A^T Ax$ which is a quadratic form with associated matrix $A^TA$. It is known that for a quadratic form and associated matrix $Q$ , the maximum value on the unit sphere is equal to the largest eigenvalue of $Q$ and the minimum value on the unit sphere is equal to the smallest eigenvalue of $Q$. If we calculate $A^TA$ we get $$ \left( \begin{matrix} 2 & 1 & 1 \\ 1 & 2 & -1 \\ 1 & -1 & 2 \\ \end{matrix}\right) $$ whose smallest eigenvalue is $0$ and largest eigenvalue is $3$. This is the minimum and maximum of $||Ax||^2$, so the minimum and maximum value of $||Ax||$ are $0$ and $\sqrt{3}$.