Given the cubic equation: $$x^3-2kx^2-4kx+k^2=0.$$
If one root of the equation is less than $1$, another root is in the interval $(1,4)$ and the third root is greater than $4$, then the value of $k$ lies in the interval $\left(a+\sqrt{b},b(a+\sqrt{6})\right)$ where $a,b\in \mathbb{N}$.
Find the value of $(a+b)^3+(ab+2)^2$.
My approach
Let the roots be $p,q,r$
So
$$p<1$$
$$1<q<4$$
$$r>4$$
$$p+q+r = 2k$$
$$pq+qr+rp = -4k$$
$$pqr = -k^2$$
Let $f(x)=x^3-2kx^2-4kx+k^2$. From the information given, $f$ has three distinct real roots. Also based on the leading term, we know $f \to \infty$ as $x \to \infty$ and $f \to -\infty$ as $x \to -\infty$.
For a root to be less than $1$, we should have $f(1) >0$. Likewise $f(4)<0$. This gives us \begin{align*} k^2-6k+1 & > 0\\ k^2-48k+64 & < 0 \end{align*} The first inequality gives $k \in (-\infty, 3-2\sqrt{2}) \cup (3+2\sqrt{2}, \infty)$. The second inequality gives $k \in (24-16\sqrt{2}, \,\, 24+16\sqrt{2})$. The intersection of these intervals gives $$k \in (3+2\sqrt{2}, \,\, 24+16\sqrt{2})=(3+\sqrt{8}, \,\, 8(3+\sqrt{8})).$$ Thus $a=3,b=8$.
Note: I think there is a possible typo in your question. It should be $k\in(a+\sqrt{b}, b(a+\color{red}{\sqrt{8}}))$