Deciding monotonic nature of $h(x)$ given another function $f(x)$

Question

Question:

If $f(x)$ is a strictly increasing function, then $h(x)=f(x)-a(f(x))^2+a(f(x))^3$ is a non-monotonic function for what set of values of $a$?

Here both $f(x)$ and $h(x)$ take only real values.

My attempt:

For $h(x)$ to be monotonic, it's derivative should either be increasing or decreasing for all $x$. So, by putting the derivative of the function as positive for all $x$ and considering the fact that $f'(x)$ is also positive, I got the set of values of a between $(0,3)$. However the answer excludes all of these values.

Please provide a detailed solution. All help is appreciated.

Answer 1

Let $$g(x)=x-ax^2+ax^3$$ Now, note that: $$h(x)=g(f(x))=(g\circ f)(x)$$ So, it is needed that $g$ is non-monotonic in order to have that $h$ is non-monotonic, since, if $g$ was either strictly increasing or decreasing, then so would $h$ be.

So, for out function $g$, let us find these values of $a$ for it is monotonic:

$$g'(x)=3ax^2-2ax+1$$ Now, this is a second degree polynomial and one can see that: $$\Delta=4a^2-12a=4a(a-3)$$ So, we need to exclude the cases when $g'$ does not change sign, so, we need that it has two distinct roots, so we need $$\Delta>0\Leftrightarrow a\in(-\infty,0)\cup(3,+\infty)$$ So, we now need to check for wich of these values of $a$ $h$ is non-monotonic. But this depends on $f$, for the following simple reason:

For any $a\in(-\infty,0)\cup(0,+\infty)$ note that $g$ is strictly increasing for every $x>x_0:=\frac{a+\sqrt{a(a-3)}}{3}$.

So, if, for instance, one chooses as $f$: $$f(x)=x_0+\arctan x$$ then, it is clear that, despite $a\in(-\infty,0)\cup(3,+\infty)$, we have that $h$ is strictly monotonous.

So, the largest possible set of values for $a$ is $$(-\infty,0)\cup(3,+\infty)$$ but, not neccessarily all of these values are appropriate - this, as stated above - depends on $f$.

However, if $f$ is onto $\mathbb{R}$ then exactly these values of $a$ are the ones requested - can you see why?