Let $p(t) = at^2 + bt + c \in \mathbb{R}[x]$ be a degree $2$ polynomial.
Let $r(x, y) = \frac{a(x, y)}{b(x, y)} \in \mathbb{R}[x, y]$ be a rational function in $x$ and $y$.
I am interested in a closed form for integrals of the form
$$\int_{t_1}^{t_2} r(t, \sqrt{p(t)}) dt$$
Does such a closed form even exist? Or if not, then perhaps there is a broad class of polynomials $p$ for which it does. For instance, if $p(t) = t^2 + c^2$ or $p(t) = t^2 - c^2$?
On
A closed form for the antiderivative (in terms of elementary functions) always exists, but the expression for the antiderivative depends on the specific functions $r$ and $p$, so probably there is no reasonable general formula in terms of arbitrary $p, r$. In particular the kinds of terms that occur are influenced by the sign of the discriminant $$\Delta := b^2 - 4 a c$$ of $p(t) = a t^2 + b t + c$. Indeed, you'll recall that the choice of function to use in the standard trigonometric substitution depends on signs of $\Delta$ and the leading coefficient $a$. But we can transform our algebraic integral $\int r(t, \sqrt{p(t)}) \,dt$ into a rational integral with a general technique, after which the usual method (factoring, applying the Method of Partial Fractions, integrating term-by-term) can in principle be applied.
In the case $a > 0$, Euler's First Substitution $$\sqrt{a t^2 + b t + c} = \sqrt{a} t + u, \qquad dt = -\frac{2 \sqrt{a} (u^2 + c) - 2 b u}{(2 \sqrt{a} u - b)^2} \,du$$ transforms our integral into the rational integral $$ -2 \int \frac{\sqrt{a} (u^2 + c) - b u}{(2 \sqrt{a} u - b)^2} r\left(-\frac{(u^2 - c)}{2 \sqrt{a} u - b}, \frac{\sqrt{a} (u^2 + c) - b u}{2 \sqrt{a} u - b}\right) \,du .$$ (An analogous substitution works in the case $a < 0$.)
Here's another approach for specific combinations of signs of $\Delta$ and $a$: Notice that in the case, e.g., $p(t) = t^2 + k^2$, $k > 0$, (so, $\Delta < 0, a > 0$), applying the standard trigonometric substitution $t = k \tan \theta, dt = k \sec^2 \theta \, d\theta$ transforms the integral to the integral $$d \int r(k \tan \theta, k \sec \theta) \sec^2 \theta \,d\theta ,$$ which is rational in the standard trigonometric functions. Then the Weierstrass substitution $\theta = 2 \arctan v, d\theta = \frac{2 \,dv}{1 + v^2}$ transforms the integral into the rational integral $$2 k \int \frac{1 + v^2}{(1 - v^2)^2} r\left(\frac{2 k v}{1 - v^2}, \frac{k (1 + v^2)}{1 - v^2}\right) \,dv .$$
I am not aware of any general closed form for such an integral. However, several special cases exist. Let, $p(t)$ be a general second-degree polynomial as you defined above. For functions of the form $r(t,\sqrt{p(t)})$, one case when closed forms exist is if $b(t,\sqrt{p(t)})=\left[a\left(t,\sqrt{p(t)}\right)\right]^2$ and $a(t,\sqrt{p(t)}):= \sqrt{p(t)}$ . We have
$$ \int\dfrac{\sqrt{at^2+bt+c}}{at^2+bt+c}\,dt = \dfrac{1}{\sqrt{a}}\,\ln\,\left|2a\sqrt{\frac{4ac-b^2}{a}}\sqrt{t\cdot\left(at+b\right)+c}+\sqrt{4ac-b^2}\left(2at+b\right)\right| + C \tag{1}$$ To obtain a closed form for the class of polynomials where $p(t)=t^2 + c^2$, you can just set $a=1$, $b=0$ and $c=c^2$ in $(1)$. This gives us
$$ \int\dfrac{\sqrt{t^2+c^2}}{t^2+c^2}\,dt = \ln\,\Bigg|c\sqrt{t^2+c^2}+ct\Bigg| +C .\tag{2} $$ I could not find one such closed form for $p(t)=t^2 -c^2$ without using complex integration.
Note that this might not be the complete list of all closed forms but this should get you started.