Let $P(x)=x^n+a_{n-1}x^{n-1}\dots+a_0$
Given $a_{n-1}^2<a_{n-2},$ prove not all roots can be real
My attempt:
Assume all roots are real, namely $R_1,R_2,\dots,R_n$
Then $-a_{n-1}=\sum_{i=1}^n R_i$
$a_{n-2}=\sum R_i\times R_j$
Now $$(R_1+R_2+\dots+R_n)^2<(R_1R_2+R_1R_3\dots R_{n-1}R_{n})\\ \implies R_1^2+R_2^2\dots R_n^2<-(R_1R_2+R_1R_3\dots R_{n-1}R_{n})$$
But we can't tell if RHS is actually negative or positive. Also this kinda looks like cauchy-schwarz. Can it be used here?
Hint: From $\displaystyle (\sum_i R_i)^2 < \sum_{i< j} R_i R_j$, we may add $\frac12 \sum R_i^2$ to both sides to get $$\left(\sum_i R_i \right)^2 + \frac12\sum_i R_i^2 < \frac12 \left(\sum_i R_i \right)^2 $$ $$\implies \frac12\left(\sum_i R_i \right)^2 + \frac12\sum_i R_i^2 < 0$$
which is absurd.