Question Let f(x)= x$^{3}+ax^{2}+bx+5sin^{2}x$ be an increasing
function$\forall$ x $\in$$\mathbb{R}$$ Then$
(a) $a^2 -3b-15$<0
(b) $a^2 -3b-15$>0
(c)$a^2 -3b+15<$0
(d)a>0,b>0
My Approach i tried by making perfect square of the derivative
$\left\{ x+\frac{a}{3}\right\} ^{2}$+$\frac{b}{3}$-$\frac{a}{9}-5\geq$0 $\left\{ sin2x\geq-1\right\} $ $\Longrightarrow$a$^{2}$-3b+45$\leq$0
but it doesn't gives the required result
Choice $(a)$ is correct.
For $a,b \in \mathbb{R},\;$let $f:\mathbb{R} \to \mathbb{R}\;$be defined by $$f(x)= x^3+ax^2+bx+5\sin^2(x)$$
Suppose $a,b$ are such that
$\qquad{\small{\bullet}}\;\;f\;$is an increasing function.
$\qquad{\small{\bullet}}\;\;a^2-3b -15 \ge 0$.
Our goal is to derive a contradiction.
Since $f$ is increasing, it follows that $f'(x) \ge 0$, for all $x \in \mathbb{R}$.
Note that \begin{align*} f'(x) &= 3x^2 + 2ax + b + 10\sin(x)\cos(x)\\[4pt] &= 3x^2 + 2ax + b + 5\sin(2x)\\[4pt] &=p(x) + 5 + 5\sin(2x)\\[4pt] \end{align*} where $p(x) = 3x^2 + 2ax + b - 5$.
By the quadratic formula, $p$ has roots $r_1,r_2$ given by \begin{align*} r_1 &= \frac{-a - \sqrt{a^2 + 3b + 15}}{3}\\[4pt] r_2 &= \frac{-a + \sqrt{a^2 + 3b + 15}}{3}\\[4pt] \end{align*} From $a^2 - 3b-15 \ge 0$, we get $a^2 - 3b + 15 \ge 30$.
It follows that $r_1,r_2$ are distinct real numbers, and $p(x) < 0$, for $x \in (r_1,r_2)$. \begin{align*} \text{Also,}\;\;r_2 - r_1 &= 2\left( \frac {\sqrt{a^2 + 3b + 15}} {3} \right)\\[5pt] &\ge \frac{2\sqrt{30}}{3}\\[5pt] & > \frac{2\sqrt{25}}{3}\\[5pt] &=\frac{10}{3}\\[5pt] & > \pi\\[4pt] \end{align*} Since $r_2-r_1 > \pi$, it follows that for some integer $n$, we have $r_1 < \frac{3\pi}{4} + n\pi < r_2$.
Then letting $t = {\large{\frac{3\pi}{4}}} + n\pi$, we get \begin{align*} f'(t) &= p(t) + 5 + 5\sin(2t)\\[4pt] &=p(t) + 5 + 5\sin\left({\small{\frac{3\pi}{2}}} + 2n\pi\right)\\[4pt] &=p(t) + 5 + 5\sin\left({\small{\frac{3\pi}{2}}}\right)\\[4pt] &=p(t) + 5 + 5(-1)\\[4pt] &=p(t)\\[4pt] &< 0\\[4pt] \end{align*} contradiction.
It follows that choice $(a)$ is correct, as claimed.
If it's given that only one of the choices $(a),(b),(c),(d)$ is correct, then since $(a)$ is correct, we're done.
If we need to show that choice $(a)$ is the only correct choice, then we have more work to do . . .
To invalidate choices $(b)$ and $(d)$, use the values $a=0,\,b=6$.
Then for all $x \in \mathbb{R}$, \begin{align*} f'(x)&=3x^2 + 2ax + b + 5\sin(2x)\\[4pt] &=3x^2 + 6 + 5\sin(2x)\\[4pt] &\ge 3x^2 + 6 + 5(-1)\\[4pt] &= 3x^2 +1\\[4pt] & > 0\\[4pt] \end{align*} so $f$ is an increasing function.
But then
To invalidate choice $(c)$, let \begin{align*} a=&{\small{\frac{\pi}{4}}}\\[4pt] b=&{\small{\frac{a^2}{3}}} +\left( 5 - {\small{\frac{1}{10}}} \right) \\[4pt] \end{align*} and verify that $f'(x) > 0\;$for all $x \in \mathbb{R}$.
The easiest verification is graphical (i.e., graph $f',\;$and see that the graph lies entirely above the $x$-axis).
It can also be verified algebraically, with a little more work.
I'll omit the algebraic verification for now, but I'll supply the (somewhat messy) details, if requested.
So for the values of $a,b,\;$as given above, $f$ is an increasing function, but $$a^2 - 3b + 15 = {\small{\frac{3}{10}}} > 0$$ hence choice $(c)$ fails.
Therefore choice $(a)$ is the only correct choice.