Let $p, q , r$ be distinct real nos such that $ap^2 + bp + c = (\sin(\theta))p^2 +(\cos(\theta))p$ similarly we get a total of three equations if we replace $q$ and $r$ in place of p. where $a, b, c$ belongs to $\Bbb{R}$
Question is find the max value of: $(a^2 + b^2)/a^2+ 3ab +5b^2$
Let's write the three equations: $$\begin{align} ap^2 + bp + c &=& (\sin(\theta))p^2 +(\cos(\theta))p \\ aq^2 + bq + c &=& (\sin(\theta))q^2 +(\cos(\theta))q \\ ar^2 + br + c &=&(\sin(\theta))r^2 +(\cos(\theta))r\end{align}$$
Now we have to keep in mind that $a,b,c,\theta$ are variables and $p,q,r$ are constants and we need to find the max value of $\frac{a^2+b^2}{a^2+3ab+5b^2}$. first we can eliminate $c$ by subtractions of the three equations, and as $p,q,r$ are distinct we can simplify by $p-q\neq 0$ and $q-r\neq 0$ to obtain: $$\begin{align} a(p+q) + b &=& (\sin(\theta))(p+q) +(\cos(\theta)) \\ a(q+r) + b &=& (\sin(\theta))(q+r) +(\cos(\theta)) \end{align}$$
and hence one can see that $a(p-r)=\sin(\theta) (p-r)$ and again we know that $p-r\neq 0$ hence $a=\sin(\theta)$ and $b=\cos(\theta)$ now it remains to find the max value of: $$\frac{a^2+b^2}{a^2+3ab+5b^2}=\frac{1}{1+3\cos(\theta)\sin(\theta)+4\cos(\theta)^2} $$
and we can write the denominator as $$1+3\cos(\theta)\sin(\theta)+4\cos(\theta)^2=\frac{1}{2}(3\sin(2\theta)+4\cos(2\theta)+6) =\frac{1}{2}(5 \sin(2\theta+\alpha)+6)$$ where $\alpha=\arctan(\frac{3}{4})$.
Finally the maximum value of the given expression is $2$ and the minimum value is $\frac{2}{11}$.
Comment The fact that $a=\sin(\theta)$ and $b=\cos(\theta)$ can be justified by the follwing argument. Let $P(x)=(a-\sin(\theta))x^2+(b-\cos(\theta))x+c$, we observe that $P$ is a polynomial of degree less that $2$ which have more than $3$ roots (p,q,r) hence $P=0$ and from here we have $a-\sin(\theta)=0$ and $b-\cos(\theta)=0$.