Consider $x^2-(a^2+b^2)x+a^2 b^2-(1-a)^2(1-b)^2=0$, where $0<a,b<1$. If $x_1,x_2$ are the roots of this equation, is $\max(|x_1|,|x_2|)<1$ always true? If not, for what values of $a,b$ it holds?
This comes from calculating the eigenvalues for a two by two matrix. The way I first intended to solve this problem was to think of three possible conditions: 1) $x_1=x_2$; 2) $x_1 \neq x_2$ and $x_1,x_2\in \mathbb{R}$ 3) $x_1 \neq x_2$ and $x_1,x_2 \in \mathbb{C}$. From the equation, we know that $x_1+x_2=a^2+b^2<2$, also we can see that $-1<x_1 x_2<1$. Continuing this yields an ugly way of solving it and I think there should be simpler ways to do this. Please let me know.
I think $\max \{|x_1|, |x_2|\} < 1$. Indeed let $$f(x) = x^2 - (a^2 + b^2)x + a^2b^2 -(1-a)^2(1-b)^2$$ First the roots of $f$ are always real the discriminant $$\Delta = (a^2 + b^2)^2 - 4a^2b^2 + 4(1-a)^2(1-b)^2 = (a^2 - b^2)^2 + 4(1-a)^2(1-b)^2 \ge 0$$ As leading coefficient is equal to $1$ ($> 0$)then for every $x$ $$f(x) \le 0 \implies |x| \le \max\{|x_1|, |x_2|\}$$ $$f(1) = 1 - a^2 - b^2 + a^2b^2 -(1-a)^2(1-b)^2 = (1-a^2) (1-b^2)-(1-a)^2(1-b)^2 = (1-a) (1-b) ((1+a)(1+b) - (1-a)(1-b)) > 0$$ Now $$f (-1) = 1 + a^2 + b^2 + a^2b^2 -(1-a)^2(1-b)^2 = (1+ a^2) (1+b^2) - (1-a)^2 (1-b)^2 > 0$$ In addition we have $0 < x_1 + x_2 = a^2 + b^2 < 2$. This implies that $f(x) > 0$ for every $x \ge 1$ or $x \le -1$.