How to find x so that $\|A x\| = \|A\| \|x\|$ holds

Question

The subbordinance property of matrix-vector multiplication states that $\|A x\| \le \|A\| \|x\|$ where $\|x\|$ is the norm of vector $x$ and $\|A\|$ is the induced norm of matrix $A$.

Many textbooks provide the proof of this result in theory, but none gives a specific way to find such an $x$ that satisfies the equality $\|Ax\| = \|A\|\|x\|$. For example, given a simple 2x2 matrix $A$, how would one find a 2-D vector $x$ that satisfies the equality?

I am actually interested in the significance of such vector $x$, in addition to its existence in theory.

The Cauchy-Schwarz inequality $|\langle x, y \rangle| \le \|x\|\|y\|$ becomes an equality if vectors $x$ and $y$ are linearly dependent, i.e., $y=cx$, or the angle between $x$ and $y$ is zero. Is there some similar interpretation for $A$ and $x$ that satisfy the equality $\|Ax\|=\|A\|\|x\|$ ?

Answer 1

In general, we have $\|A\|^2=\|A^* A\|$. Furthermore, $A^*A\geq 0$, so there is a unit vector $x$ such that $$\|Ax\|^2=\langle x,A ^*Ax\rangle=\|A^*A\|=\|A\|^2.$$ In fact, any unit eigenvector $x$ of $A^*A$ corresponding to the largest eigenvalue of $A^*A$ will work.

To clarify, this is with respect to the operator norm induced by the 'standard' norm on $\mathbb C^n$ which makes that space into a Hilbert space (in order that we may easily speak of the adjoint of an operator).

Answer 2

Thanks to everyone who contributed to this discussion. Here is what I got:

Consider

||Ax||^2=<(Ax),(Ax)>=x^*(A^*A)x

if x is the eigenvector corresponding to the greatest eigenvalue of A^*A, which is the spectrum norm ||A||_2 squared, i.e., (A^*A) x=||A||^2 x, then the above becomes

||Ax||^2=x^*||A||^2 x=||A||^2 =(||A|| ||x||)^2

i.e.,

||Ax||=||A|| ||x||

However, this result is only valid for the spectrum norm ||A||_2, the greatest singular value of A (square root of the greatest eigenvalue of A^*A). It would be nice to know how to find such an x for the equality to hold for other forms of norm of A, such as ||A||_1 and ||A||_int, if such methods exist.

Answer 3

We have to maximize $$\|A{\bf x}\|^2=\langle A{\bf x}, A{\bf x}\rangle=\langle A^*A\,{\bf x},{\bf x}\rangle$$ subject to the condition $\|x\|^2=1$. The map $A^*A$ , resp. its matrix $[A^*A]$, is symmetric positive definite. Therefore there exists an orthonormal basis of ${\mathbb R}^n$ with $$[A^*A]={\rm diag}(\lambda_1,\lambda_2,\ldots,\lambda_n),\qquad \lambda_1\geq\lambda_2\geq\ldots\geq\lambda_n\geq0\ .$$ Using this basis we have to maximize $$\sum_{k=1}^n\lambda_k x_k^2$$ subject to the condition $x_1^2+x_2^2+\ldots+x_n^2=1$. The maximum obviously has the value $\lambda_1$, and is taken, e.g., at the vector ${\bf e}:=(1,0,\ldots,0)$. It follows that $$\|A{\bf x}\|=\sqrt{\|A{\bf x}\|^2}\leq\sqrt{\lambda_1}\qquad(\|{\bf x}\|=1)\ ;$$ and on the other hand we have $\|A{\bf e}\|=\sqrt{\lambda_1}$.

When $$A=\left[\matrix{a&b\cr c&d\cr}\right]$$ we can write a general unit vector in the form ${\bf x}=(\cos\phi,\sin\phi)$, and then have to maximize $$\eqalign{\|A{\bf x}\|^2&=(a^2+c^2)\cos^2\phi+2(ab+cd)\cos\phi\sin\phi+(b^2+d^2)\sin^2\phi\cr &={a^2+c^2+b^2+d^2\over2}+{a^2+c^2-b^2-d^2\over2}\cos(2\phi)+(ab+cd)\sin(2\phi)\ .\cr}$$ A well known formula for the amplitude of a trigonometric binomial then gives $$\lambda_1={a^2+c^2+b^2+d^2\over2}+{1\over2}\sqrt{(a^2+c^2-b^2-d^2)^2+4(ab+cd)^2}\ .$$

Answer 4

For the $2$-norm, pick $x$ to be a singular vector corresponding to the largest singular value of $A$.

The SVD is visualized in this picture, which illustrates the fact that $A$ maps a circle to an ellipse (or a sphere to an ellipsoid). The singular values and singular vectors of $A$ satisfy: \begin{equation*} A v_i = \sigma_i u_i. \end{equation*} The picture makes it clear that the $2$-norm of $A$ is $\sigma_1$, the largest singular value of $A$. So \begin{align*} \| A v_1 \|_2 &= \| \sigma_1 u_1 \|_2 \\ &= \sigma_1 \\ &= \|A\|_2 \\ &= \|A \|_2 \|v_1\|_2. \end{align*} (Note that the singular vectors are all unit vectors.)

This is equivalent to some of the other answers, which don't mention the SVD explicitly.