Let $A$ be a diagonal positive matrix, $B$ is a hermitian symmetry matrix. Is there a method to find $x$ such that $x = \arg \min_{|x|=1} (x^H A x)(x^H B x)$
I am not sure that whether the following is correct, it might be equivalent to a Rayleigh quotient problem: $x = \arg \min_x \frac {x^H B x}{x^H A^{-1} x}$
Am I right?
Your question was stated in terms of the unit vector $x$, but let me reformulate it in terms of an unconstrained vector $y$ defined as $$ \eqalign { \lambda x &= y \cr \lambda^2 &= y^Hy = y^*:y \cr 2\lambda\,d\lambda &= y^*:dy \cr } $$ where colon denotes the matrix trace product, i.e. $$A:B={\rm tr}(A^TB)$$
For convenience, let me define two scalars $$ \eqalign { \alpha &= y^HAy = {\rm tr}(y^HAy) = A^Ty^*:y &\,\,\implies d\alpha = A^Ty^*:dy \cr \beta &= y^HBy= {\rm tr}(y^HBy) = B^Ty^*:y &\,\,\implies d\beta = B^Ty^*:dy \cr } $$ Note that $(y,y^*)$ will be treated as formally independent variables. You can include the contributions from $dy^*$ in the calculations, but it won't change the final result and makes the intermediate expressions twice as lengthy.
Now write the cost function in terms of these new variables and find its differential & gradient wrt $y$ $$ \eqalign { Q &= \lambda^{-4}\alpha\beta \cr dQ &= \lambda^{-4}\beta\,d\alpha + \lambda^{-4}\alpha\,d\beta + \alpha\beta\,d\lambda^{-4} \cr &= \lambda^{-4}\beta\,A^Ty^*:dy + \lambda^{-4}\alpha\,B^Ty^*:dy - 4\alpha\beta\lambda^{-6}\,\lambda d\lambda \cr &= \lambda^{-4}\Big(\beta\,A^T + \alpha\,B^T - 2\lambda^{-2}\alpha\beta I\Big)y^*:dy \cr \frac{\partial Q}{\partial y} &= \lambda^{-4}\Big(\beta\,A^T + \alpha\,B^T - 2\lambda^{-2}\alpha\beta I\Big)y^* \cr } $$ Setting the gradient to zero and solving for $y$ yields $$ \eqalign { y^* &= \frac{\lambda^2}{2\alpha\beta}\Big(\beta\,A + \alpha\,B\Big)^T y^* \cr y &= \mu\,(\beta A + \alpha B)^H y \cr\cr } $$ This is not a simple eigenvalue equation, since $\,(\alpha, \beta, \mu)\,$ on the RHS are functions of the vector $y$ on the LHS. So it's really just one step in an iterative process to calculate $x$ $$ \eqalign { x_+ &= \frac{(\beta A+\alpha B)^Hx}{\|(\beta A+\alpha B)^Hx\|} \cr } $$