If $A\ge0$ and $AB$ is self-dual, then $|\langle ABx,x\rangle| \le \Vert B\Vert_2 \langle Ax,x\rangle$

Question

Claim to prove (exercise 91.7 of Halmos 1958):

Let $A$ and $B$ be linear transformations of a finite-dimensional real or complex inner product space. If $A$ is positive semi-definite and $AB$ is self-dual (i.e. Hermitian or symmetric in the real or complex case), then $|\langle ABx,x\rangle| \le \Vert B\Vert_2 \cdot \langle Ax,x\rangle$.

Note: The absolute value in the claim above was omitted by mistake in the first version of the question. The proof in the accepted answer still works mutatis mutandis.

---

I really haven't made much progress (and I feel like it should be fairly simple), but here my observations for what it's worth. Each path seems like a dead end...

Clearly $|\langle ABx,x\rangle| \le \Vert B\Vert \Vert x\Vert \Vert Ax\Vert$ and $\langle Ax,x\rangle \le \Vert x\Vert \Vert Ax\Vert$.

By the spectral theorem, $A$ and $AB$ both have an orthogonal basis of eigenvectors with real eigenvalues. Say, $\{e_i\}_{i=1}^n$ are orthonormal eigenvectors of $A$ with eigenvalues $\sigma_i>0$, and $\{f_i\}_i$ are orthonormal eigenvectors of $AB$ with eigenvalues $\lambda_i\in\Bbb R$. Then I can write for example $$ \langle ABx, x \rangle = \sum_{i} \lambda_i|\langle x, f_i\rangle|^2 = \sum_{i,j} \lambda_i\langle x, f_i\rangle\langle f_i, e_j\rangle\langle e_j, x\rangle, $$ $$ \langle Ax, x \rangle = \sum_{i} \sigma_i|\langle x, e_i\rangle|^2, $$ or several other variations of the above. The trouble with this approach is connecting it to $\Vert B\Vert$. I do have a characterization of what $B$ must look like for $AB$ to be self-dual. Let $T$ be the orthogonal matrix given by $Te_i = f_i$. Then let $B=CT^{-1} = CT^*$. We have $$ \langle Ce_i,e_j\rangle = \langle Bf_i,e_j\rangle = \frac{\lambda_i}{\sigma_j}\langle f_i,e_j\rangle, \quad\textrm{if }\sigma_j\ne0. $$ Also, $$ \langle Te_i,e_j\rangle = \langle e_i,f_j\rangle = \frac{\sigma_i}{\lambda_j}\langle e_i,Bf_j\rangle = 0 \quad\textrm{if }\sigma_i=0\land\lambda_j\ne0. $$ These are conditions on the entries of the matrices $C$ and $T$, and together with $T=T^*$, they completely characterize the possible matrices $B$.

Answer 1

It is equivalent to prove that $A\|B\|-AB$ is positive semidefinite (since it is self-dual).

If $A$ is invertible, then it is equivalent to $A^{1/2}(\|B\|I -B)A^{1/2}$ has all nonnegative eigenvalues, and to prove it, it is sufficient that $x^*(\|B\|I-B)x\ge 0$ for every $x$, that is true for easy reasons.

If $A$ is not invertible, you can project onto its rank and reduce to the previous case. In fact, take the EVD $A=QDQ^*$ and call $C=Q^*BQ$. Notice that $DC=C^*D$ and the thesis becomes the same with $D,C$ instead of $A,B$ and $D$ is diagonal. Suppose WLOG that the zeros eigenvalues are all at the bottom of $D$, and they are $k$. Since $DC$ has the last $k$ rows equal to zero and it's self-adjoint, you can conclude that $DC$ has also the last $k$ columns equal to zero and thus $C = \begin{pmatrix}E&0\\*&*\end{pmatrix}$ where the zero submatrix has dimension $(n-k)\times k$. As a consequence, if $F$ is the maximal non-singular diagonal principal submatrix of $D$, $$x^*DCx = y^*FEy \qquad \|C\| x^*Dx \ge \|E\| y^*Fy $$ where $y$ is the truncation of $x$ and $F$ is now invertible, so we can reduce to the previous case.

Answer 2

By a change of orthonormal basis we may assume that $A=\pmatrix{A_1&0\\ 0&0}$ for some positive definite matrix $A_1$. Since $AB$ is self-adjoint, $B$ must be in the form of $\pmatrix{B_1&0\\ B_2&B_3}$ when partitioned conformably, with $A_1B_1$ being self-adjoint. As $\|B_1\|_2\le\|B\|_2$, it suffices to prove that $$ |\langle A_1B_1v,v\rangle|\le\|B_1\|_2\langle A_1v,v\rangle $$ for every vector $v$.

In other words, it suffices to prove the original inequality under the additional assumption that $A$ is positive definite. In this case, let $H=AB$, $|H|=\sqrt{H^2}$ and $y=A^{1/2}x$. Note that $-|H|\le H\le |H|$ and $|H|=HU$ for some unitary matrix $U$. Therefore $$ \begin{aligned} |\langle ABx,x\rangle| &=|\langle Hx,x\rangle|\\ &\le\langle |H|x,x\rangle\\ &=\langle|H|A^{-1/2}y, A^{-1/2}y\rangle\\ &=\langle A^{-1/2}|H|A^{-1/2}y,y\rangle\\ &\le\rho(A^{-1/2}|H|A^{-1/2})\|y\|_2^2\\ &=\rho(A^{-1}|H|)\langle Ax,x\rangle\\ &\le\|A^{-1}|H|\|_2\langle Ax,x\rangle\\ &=\|A^{-1}HU^\ast\|_2\langle Ax,x\rangle\\ &=\|A^{-1}H\|_2\langle Ax,x\rangle\\ &=\|B\|_2\langle Ax,x\rangle.\\ \end{aligned} $$