I have a general quadratic equation with complex coefficient
$$ax^2+bx+c=0$$
where a, b c are all complex numbers.
I wonder is there a necessary and sufficient condition to guarantee that all roots of this quadratic equation has magnitude less than 1?
Any sufficient condition would also be helpful.
The following assumes $\,a \ne 0\,$, $\,c \ne 0\,$ since the other cases are trivial.
Let $\,\dfrac{b}{a} = - 2 \beta\,$, $\,\dfrac{c}{a}= \gamma\,$, then after dividing by $\,a\,$ the equation becomes $\,x^2 - 2 \beta x + \gamma = 0\,$, and the roots can be written as $\,x_{1,2} = \beta \pm \delta\,$ where $\,\delta^2 = \beta^2 - \gamma\,$ is the reduced discriminant.
The roots $\,\beta \pm \delta\,$ are the diagonals of the parallelogram with sides $\,\beta, \delta\,$. By the parallelogram law:
$$ |\beta-\delta|^2 + |\beta+\delta|^2 = 2\left(|\beta|^2 + |\delta|^2\right) \tag{1} $$
Let $\,s \gt 0\,$ be the magnitude of one of the roots, then the magnitude of the other one is $\,\dfrac{|\gamma|}{s}\,$ by Vieta's relations. Substituting in $\,(1)\,$:
$$ s^2 + \frac{|\gamma|^2}{s^2} = 2\left(|\beta|^2 + |\delta|^2\right) \tag{2} $$
It follows that $\,s\,$ is one of the positive roots of the biquadratic with real coefficients:
$$ s^4 - 2\left(|\beta|^2 + |\delta|^2\right) s^2 + |\gamma|^2 = 0 \tag{3} $$
With $\,s^2 = t\,$, $\,|\beta|^2 + |\delta|^2 = u \gt 0\,$, $\,|\gamma|^2 = v \gt 0\,$, the equation is a real quadratic in $\,t\,$:
$$ t^2 - 2 u t + v = 0 \tag{4} $$
For verification, the latter has two positive real roots (as expected), since their sum and product are both positive, and its reduced discriminant is non-negative:
$$ u^2 - v = \left(|\beta|^2 + |\delta|^2\right)^2 - |\gamma|^2 = \underbrace{\left(|\beta|^2 + |\delta|^2 - |\gamma|\right)}_{\ge\, 0\, \text{ since } \, |\gamma| \,\le\, |\beta|^2 + |\beta^2 - \gamma|\\ \text{by the triangle inequality}}\; \underbrace{\left(|\beta|^2 + |\delta|^2 + |\gamma|\right)}_{\gt \, 0} \;\ge\; 0 \tag{5} $$
Finally, the roots $\,x_{1,2}\,$ are inside the unit circle iff the roots of $\,(4)\,$ are in $\,(0,1)\,$. Since the roots $\,u \pm \sqrt{u^2 - v}\,$ are known to be positive, the condition reduces to the larger one being less than $\,1\,$:
$$ u + \sqrt{u^2 - v} \,\lt\, 1 \tag{6} $$
Equivalently:
$u \lt 1 \tag{6.a}$
$\require{cancel} u^2 - v \lt (1-u)^2 \;\;\iff\;\; \cancel{u^2} - v \lt 1 - 2 u + \cancel{u^2} \;\;\iff\;\; 2 u \lt 1 + v \tag{6.b}$
Since $\,v \le u^2\,$ by $\,(5)\,$ and $\,u \lt 1\,$ by $\,(6.a)\,$ it follows that $\,v \lt 1\,$, which just confirms that $\,|x_{1,2}| \lt 1\,$ $\,\implies |x_1x_2| = |\gamma| \lt 1\,$ $\,\implies v \lt 1\,$. Then $\,(6.a)+(6.b)\,$ can be coalesced into:
$$ 2 u - 1 \;\lt\; v \;\lt\; 1 \tag{7} $$
Or, reverting to the $\,\beta, \gamma\,$ variables:
$$ 2 \left(|\beta|^2 + |\beta^2 - \gamma|\right) - 1 \;\lt\; |\gamma|^2 \;\lt\; 1 \tag{8} $$
Or, reverting to the $\,a,b,c\,$ variables:
$$ |b|^2 + |b^2 - 4ac| - 2\,|a|^2 \;\lt\; 2\, |c|^2 \;\lt\; 2 \tag{9} $$