Say we're in a (complex) Hilbert space $H$ where we're given some two elements $a,b\in H$ of norm 1.
The question is to find an element $\psi\in H$ (of norm 1) that maximizes $|\langle \psi,a\rangle \langle b,\psi\rangle|$ (or $|\langle \psi,a\rangle \langle b,\psi\rangle|^2$ if you prefer).
Mainly what I want to prove is one of the following inequalities: $$ |\langle \psi,a\rangle \langle b,\psi\rangle|\leq |\langle a,b \rangle| $$ $$ |\langle \psi,a\rangle \langle b,\psi\rangle|\leq\bigg|\frac{1+|\langle a,b\rangle| }{2}\bigg| $$
Intuitively, my choice is to say that this maximizes when $\psi = \frac{a+b}{2}$, modulo the norm. And in fact one can get close to proving the inequalities by plugging that in. But I can't prove why this should be the case.
It is obvious that $\psi$ should be in the span of $a,b$, since any component that's outside of this subspace will be wasted. So we can write $\psi = xa + yb$ for some complex $x$ and $y$. But $a$ and $b$ need not be orthogonal, so it's not so simple.
Is there an inequality that one can use here to prove one of the above statements, or a quick way to find $\psi$?
What you consider here is the numerical radius of the rank-1 operator $A:H\to H$, $h\mapsto \langle b,h\rangle a$. To put things into context, the numerical range of $A$ is given by $$ W(A)\overset{\text{Def.}}=\lbrace\langle \psi,A\psi\rangle\,|\,\psi\in H,\|\psi\|_H=1\rbrace=\lbrace\langle \psi,a\rangle\langle b,\psi\rangle\,|\,\psi\in H,\|\psi\|_H=1\rbrace $$ and the numerical radius of $A$ by $$ r(A)=\sup\lbrace |\lambda|\,|\,\lambda\in W(A)\rbrace=\sup_{\psi\in H,\|\psi\|_H=1}|\langle \psi,a\rangle\langle b,\psi\rangle|\,. $$ As $A$ is of rank one, Lemma 2.1 in this paper (click here for accessible pdf) yields $$ r(A)=\frac{\|A\|+|\operatorname{tr}(A)|}2\,. $$ One readily verifies that $A$ has operator norm $1$ and $\operatorname{tr}(A)=\langle b,a\rangle$ so $$ |\langle\psi,a\rangle\langle b,\psi\rangle|\leq \sup_{\psi\in H,\|\psi\|_H=1}|\langle \psi,a\rangle\langle b,\psi\rangle|=r(A)=\frac{1+|\langle a,b\rangle|}2 $$ for all $\psi\in H,\|\psi\|_H=1$ which is the desired inequality.