Let $A$ and $B$ be linear transformetions of a finite-dimensional inner product space $V$. Let $A$ be positive semi-definite and assume that $AB$ is self-adjoint. Then what is the smallest $k\ge0$ such that, for all $x\in V$, $$ |\langle ABx,x\rangle| \le k\langle Ax,x\rangle? $$
For context, this is inspired by my earlier question, If $A\ge0$ and $AB$ is self-dual, then $|\langle ABx,x\rangle| \le \Vert B\Vert_2 \langle Ax,x\rangle$, which made me curious about how tight we can make the inequality. I post a complete answer myself.
(Next questions for the curious could be for example to consider what happens if we loosen or remove the assumptions on $A$ and $B$. Or maybe consider infinite-dimensional spaces, or...).
Let $P$ be the orthogonal projection on the range of $A$. (And let's just assume $A\ne0$). Let $\rho(\cdot)$ denote spectral radius, i.e. the largest eigenvalue in absolute value of the given transformation. The the smallest $k$ satisfying the inequality is $\rho(PB)$, and this bound tight.
Invertible case:
Assume that $A$ is invertible. Note first, for later, that $B$ is similar to the self-dual $A^{\frac12}BA^{-\frac12}$, so $B$ has real spectrum. (That will also be evident in a bit, but is nice to notice separately).
$|\langle ABx,x\rangle| \le k\langle Ax,x\rangle$ holds for all $x$ if and only if $A(k\pm B)\ge 0$. This is equivalent to $A^{\frac12}(k\pm B)A^{-\frac12}\ge 0$, from multiplying the self-dual and invertible $A^{-\frac12}$ on either side, which again is equivalent to all eigenvalues of $A^{\frac12}(k\pm B)A^{-\frac12}$ being non-negative. By similarity, $A^{\frac12}(k\pm B)A^{-\frac12}$ has the same eigenvalues as $k\pm B$. Finally, the eigenvalues of $k\pm B$ are non-negative if and only if $k\ge \rho(B)$.
General case:
We will reduce to the invertible case. Note that, by normality of $A$ (e.g. by the spectral theorem), we have $AP=PA=A$. Using this and $AB=B^*A$, it can easily be checked that
$$ \langle ABx,x\rangle = \langle A(PB)(Px),Px\rangle, \quad \textrm{and} \quad \langle Ax,x\rangle = \langle A(Px),(Px)\rangle. $$
Note that the range of $A$, call it $R(A)$, is invariant under $A$ and $PB$, so we can consider the restrictions of these transformations on $R(A)$. Let's denote the restriction to $R(A)$ with a prime. Then,
$$\begin{split} &\forall x\in V :|\langle ABx,x\rangle| \le k\langle Ax,x\rangle \\ \iff &\forall x\in V :|\langle A(PB)(Px),(Px)\rangle| \le k\langle APx,Px\rangle \\ \iff &\forall y\in R(A) :|\langle A'(PB)'y,y\rangle| \le k\langle A'y,y\rangle. \end{split}$$
Now, $A'$ is positive definite and $A'(PB)'$ is self-dual. By the invertible case, $k$ is minimized with $\rho((PB)') = \rho(PB)$.
For that last equality, note that $PB = (PB)|_{R(A)} \oplus (PB)|_{\ker(A)} = (PB)' \oplus 0$ has the same eigenvalues as $(PB)'$ as well as possibly $0$.