Given a unit vector $x=[1~~0~~0~\cdots~0]^\top\in\mathbb{R}^n$ and positive definite matrices $A, B\in\mathbb{R}^{n\times n}$.
My question is How could we find the upper bound of $x^\top ABA x$ in terms of $x^\top B x$ and a function of $A$, that is $$x^\top ABA x\leq f(A)x^\top Bx,$$ where $f(A)$ could be any function related to $A$, e.g., the norm of $A$.
No such $f$ exists. E.g. suppose $t>0$, $$ A=\pmatrix{2&1\\ 1&2},\,B=B(t)=\pmatrix{1&0\\ 0&t^2} \ \text{ and }\ x=\pmatrix{0\\ t^{-1}}. $$ Then $x^TBx=1$ and \begin{aligned} q(A,B(t),x):=\frac{x^TABAx}{x^TBx}&=x^TABAx=\|B^{1/2}Ax\|_2^2\\ &=\left\|\pmatrix{1&0\\ 0&t}\pmatrix{2&1\\ 1&2}\pmatrix{0\\ t^{-1}}\right\|_{\,2}^{\,2}\\ &=\left\|\pmatrix{1&0\\ 0&t}\pmatrix{t^{-1}\\ 2t^{-1}}\right\|_{\,2}^{\,2}\\ &=\left\|\pmatrix{t^{-1}\\ 2}\right\|_{\,2}^{\,2}\\ &=t^{-2}+4. \end{aligned} Since $q(A,B(t),x)\to\infty$ as $t\to0^+$, it is not bounded above by any function of $A$.
However, there is an upper bound in terms of both $A$ and $B$. When $x\ne0$, if we put $u=\dfrac{B^{1/2}x}{\|B^{1/2}x\|_2}$, we have $$ \dfrac{x^TABAx}{x^TBx}=u^TB^{-1/2}ABAB^{-1/2}u\le \rho(B^{-1/2}ABAB^{-1/2})=\|B^{1/2}AB^{-1/2}\|_2^2. $$