I have struggle on finding this function g(x).
Assume function $f(x) = 5x^3 -20x + 3$ and it is specified to find root in [0, 1]. So I guess, first thing is to find function g(x).
$$g_1(x) = \sqrt[3]{(4x - \frac{3}{5})}$$
$$|g'(x)| = \left| \frac {4} {3\sqrt[3]{(4x-\frac 3 5)^2}}\right| \le \frac 4 3 \gt 1$$
also $f(x) = x(5x^2-20)+3$ and $$g_2(x) = -\frac {3} {5x^2 - 20}$$
but I do not know which one to choose. My teacher said something like
$$|g'(x)| \lt 1 $$
So which one should I choose and why, or $g_1(x)$ and $g_2(x)$ are not good.
These are not the only choices. In fact, any function $g(x)=k f(x) + x$ would meet the fixed point condition. The most obvious for me is $g_3(x)=\frac{1}{20} ( 5x^3 + 3)$ where it is easy to check the convergence criterium $|g'(x)|<1$.