Is $|g'(x)|<1\ \forall x\in(a,b)$ is one of the hypothesis of the Fixed-Point Theorem?
The answer is NO. Can someone please enlightened me about this? My teacher reason is this...
Note that the condition must be $|g'(x)| \leq k < 1\ \forall x\in(a,b)$. This condition is equivalent to $g'(x)\in[-k,k]\ \forall x\in(a,b)$. The condition $|g'(x)|<1\ \forall x\in(a,b)$ is equivalent to $g'(x)\in(-1,1)\ \forall x\in(a,b)$. Observe that the two conditions are not the same.
On
Well, the reason is implicitly covered by the answers above. The underlying theoretical reason (if you know about Cauchy sequences), is that to be sure that the sequence $(p_n)$ is Cauchy (and hence definitely convergent), you need the constant $k$ in the formula $|g(x) - g(y)| \leq k|x-y|$ (for all $x,y \in [a,b]$) to be definitely less than one.
The basic idea is that for large $r$ and $s$ with $s >r$, we have $|p_s - p_r| \leq k^{r}(1 + k + \ldots + k^{s-r-1})|p_1 -p_0|$, and we need to be sure that the partial sums of the geometric series converge, and that $k^r \to 0$, so we need $k$ strictly less than one. The condition on derivatives allows the Mean Value Theorem to be applied to get $|g(x)-g(y)| \leq k|x-y|$ for all $x,y \in [a,b]$, as others have already commented.
On
People gave reasons why it is not true if you consider the two different hypothesis... but I will tell why the two hypothesis are not the same!
If you say that $|g'(x)| < 1$, that is saying that $g'(x)$ lies in the interval $(-1,1)$, but might be arbitrarily close to $-1$ or $1$. On the other hand, the second statement says that $|g'(x)| \le k < 1$, which means $g'(x)$ $\textbf{cannot}$ get arbitrarily close to $-1$ or $1$, but is somewhat trapped in a compact interval which is itself in $(-1,1)$. In that sense it is a stronger statement than the one you suggested (i.e. that $|g'(x)| < 1$).
I felt like this was one of the points you were missing so I mentioned it.
Let $K$ be an closed set and $g \in C^1(K), g(K) \subseteq K$ with $|g'(x)|<1\ \forall x\in K$. Then $g$ doesn't neccessiarily have a fixed point. Look for example at $g(x) = x+1/x$ on $[1,\infty)$. Then $|g'(x)|=1 - 1/x^2<1$ but you have no fixed point. I hope this is what you are looking for. If you can otherwise find a $\lambda$ such that $|g'(x)|<\lambda<1\ \forall x\in K$ then you can use the convergence of the geometric sum to show there is a fixed point.