The given function is $f(x)=x^3-3ax^2+3bx-2$.
I am aware that monotonously increasing means to continuously increase, so I tried getting this function's derivative and then setting it to zero, but to my dismay, even when $b=a$, $3x^2-6ax+3b$ isn't factorable.
I took the quadratic equation and got $1+\sqrt{2}$ and $1-\sqrt{2}$ as its roots.
The answer is $0<a<1$. I know this because I got it from the answer sheet.
All help is appreciated!
On
I prefer a straightforward calculation based on analysis of quadratic function. $$\displaylines{f'(x)=3x^2-6ax+3b=3(x^2-2ax+b)\\ =3[(x-a)^2+b-a^2] }$$ The minimal value of $f'(x)$ is attained at $x=a$ $$f'(x)\ge f'(a)=b-a^2$$ Therefore $f'(x)\ge 0$ iff $f'(a)\ge 0,$ i.e. $b-a^2\ge 0.$ In the case $a=b$ we get $a-a^2=a(1-a)\ge 0$ i.e. $0\le a\le 1.$
Slight typo: I assume you mean monotonically increasing.
Note that a strictly increasing function means its derivative is greater than or equal to 0, not equal to zero as you imply. You correctly calculated the derivative, so $3x^2-6ax+2a\ge0$.
So now our problem is to find a such that for any x, $3x^2-6ax+2a\ge0$. To do this, we will first find the x that minimizes $3x^2-6ax+2a$, because if the minimum is greater than or equal to zero, so will all other values. Taking the derivative and setting to zero, $6x-6a=0$, so $x=a$.
Substituting this back into our first inequality, we get $3a^2-6a^2+3a\ge0$, so $-a^2+a\ge0$ or $a(1-a)\ge0$. This gives us critical points of 0 and 1, and interval testing values for each region on the number line (e.g., test -1, 0.5, and 2) gives us the desired result of $0\le a\le1$.
Another way we could have solved this is by using the discriminant. When we got the inequality $3x^2-6ax+2a\ge0$, we could have observed that this means $3x^2-6ax+2a$ has at most 1 zero, so its discriminant $b^2-4ac = (6a)^2-4(3)(2a) \ge 0$. Simplifying, this gives us $a(1-a) \ge 0$ again, and so the same final answer.