Let $z_1,z_2$ be the roots of $az^2+bz+c=0;\,\,a,b,c\in \mathbb{C},\,\,a\ne 0$. Find necessary and sufficient condition for which $\max\{|z_1|,|z_2|\}<1$.
My progress:
$z_1+z_2=-\frac ba\implies |b|<2|a|$ ---(1)
and
$|z_1z_2|=\frac ca\implies |c|<|a|$ ---(2)
combining (1) and (2), we get necessary condition $|a|>\max\{\frac12|b|,|c|\}$
On internet I also got following:
(Not a complete answer, but too long for a comment. The question appears to be fully answered in the linked paper, so the following is more about a plausible approach than the final result.)
Assuming $abc \ne 0$ otherwise the quadratic equation reduces to linear ones. Then, dividing by $\,a\,$: $$ z^2 + b'z + c' = 0 \quad \text{where} \quad b'= b/a\,, \; c'= c/a $$
Let $\,d=(c')^{\,1/2}\,$, then making the substitution $\,z=dw\,$ and dividing by $\,d^2=c'\,$: $$ w^2 + b''w + 1 = 0 \quad \text{where} \quad b''= b'/c' $$ $$ (w-u)^2 = 1 - u^2 \quad \text{where} \quad u = - b''/2 $$ $$ w_{1,2} = u \pm v \quad \text{where} \quad v = \left(1-u^2\right)^{1/2} $$
The question then reduces to determining the condition for both roots $\,|w_{1,2}| \lt 1 / |d|\,$, which is: $$ \frac{1}{|d|^2} \;\gt\; \left(u \pm v\right) \cdot \left(\bar u \pm \bar v\right) =\; |u|^2 + |v|^2 \pm 2\,\text{Re}\left(u \bar v\right) $$ $$ \frac{1}{|d|^2} \;\gt\; |u|^2 + |v|^2 + 2\,\big|\text{Re}\left(u \bar v\right)\big| $$