Lets say I have $f(x)=x+a$ and $g(x)=x+b$, where $a$ and $b$ are constants. And I have to find $(gf)^n (X)$. Here I am confused with $(gf)^n$ part.
Is it correct to do it this way: $$$$ 1) $f(x)=x+a$ $$$$ 2) $g(f(x))=x+a+b$ $$$$ 3) $f(g(f(x)))=x+2a+b$ $$$$ ... $$$$ n) $g(f(...))=....$ $$$$
Your text suggests that juxtaposition and exponents refer to function composition. Let $h:=g\circ f$. Then $h:\> x\mapsto x+a+b$ is translation by the amount $a+b$, hence $h^{\circ n}$ is translation by $n(a+b)$: $$h^{\circ n}:\quad x\mapsto x+n(a+b)\ .$$ If, however, the term $(gf)(x)$ denotes the ordinary product $$g(x)f(x)=x^2+(a+b)x+ab=:p(x)$$ then $(gf)^n$ has to be interpreted as $$(gf)^n(x)=p^n(x)=\bigl(x^2+(a+b)x+ab\bigr)^n\ .$$ Ask your professor which of the two (s)he had in mind.
(The notation $h^{\circ n}$ is widely used in cases that might lead to a misunderstanding.)