the quadratic equation whose roots are a and b where $a^2 +b^2=5$ and $3(a^5+b^5)=11(a^3+b^3)$
What I Tried
$a^2 +b^2=5$
$(a+b)^2-2ab=5$
$(\text{sum of roots})^2 -2(\text{products of roots})=5$
On
Hint: If $a+b = 0$, then you have $a = \pm \sqrt{\frac{5}{2}}, b = \mp \sqrt{\frac{5}{2}}$. From this you can find the equation. If $a + b \neq 0\implies 3(a^4-a^3b+a^2b^2-ab^3+b^4)=11(a^2-ab+b^2)= 11(5-ab)$. Using $a^4+b^4 = (a^2+b^2)^2 - 2a^2b^2 = 25 - 2(ab)^2, a^3b + ab^3 = ab(a^2+b^2) = 5ab$. Thus you do have a quadratic equation in $ab$, and once you solve for $ab$, then you can solve for $a,b$ and you are done !
There is no single answer here. It turns out there are three quadratics with real coefficients and two more with complex coefficients, the roots of which satisfy the given conditions.
The quadratic equation with roots $a$ and $b$ is $x^2-sx+p$, where $s=a+b$ and $p=ab$. Some careful algebra tells us
$$a^2+b^2=s^2-2p,\quad a^3+b^2=s^3-3sp,\quad\text{and}\quad a^5+b^5=s^5-5s^3p+5sp^2$$
If $3(a^5+b^5)=11(a^3+b^3)$, then
$$3s^5-11s^3-(15s^3-33s)p+15sp^2=0$$
Now if $a^2+b^2=5$, then $2p=s^2-5$, and the equation above (multiplied by $4$) becomes
$$12s^5-44s^3-2(15s^3-33s)(s^2-5)+15s(s^2-5)^2=0$$
which simplifies to
$$0=3s^5-22s^3+45s=s(s^2-9)(3s^2+5)$$
The real roots here are $s=0$, $3$, and $-3$, and these give values $p=-5/2$, $2$, and $2$, respectively, for quadratics
$$x^2-5/2,\quad x^2-3x+2,\quad\text{and}\quad x^2+3x+2$$
The complex roots are $s=\pm\sqrt{5/3}i$, both of which give $p=-10/3$, for quadratics
$$x^2-\sqrt{5/3}ix-10/3\quad\text{and}\quad x^2+\sqrt{5/3}ix-10/3$$
Remark: Any conditions of the form $a^2+b^2=A$, $B(a^5+b^5)=C(a^3+b^3)$, will lead to a quintic in $s$ with $s$ as one factor and a quadratic in $s^2$ as another. Presumably the problem at hand was designed so that $\{a,b\}=\{1,2\}$ and $\{-1,-2\}$ are among the roots that satisfy the given conditions.