In proving that \begin{align} \lim\limits_{n\to \infty} \left[\int^{b}_{a}f^{n}(x)dx\right]^{\frac{1}{n}}=M\end{align} where $M=\max \{f(x):x\in[a,b]\}$ and $f:[a,b]\to\Bbb{R}$ is nonnegative and continuous.$
If $f(x)\leq M,\;\forall\,x\in[a,b]$, I know that $(f(x))^{n}\leq M^n,\;\forall\,x\in[a,b]?$ but do we have that $f^{n}(x)\leq M^n,\;\forall\,x\in[a,b]?$
By $f^{n}(x)\leq M^n,\;\forall\,x\in[a,b]$, I mean "composition in $n$ times". So, in essence, I'm asking: Is \begin{align}(f(x))^n=f^n(x),\;\forall\,x\in[a,b]?\end{align} Thanks!
First of all, let me point out that the conditions of $f$ is very problematic. To make sense of multiple composition of $f$, you require the range of $f$ is in $[a,b]$. But, if the interval $[a,b]$ lies completely in the negative region, say $a=-2$, $b=-1$, you cannot simultaneously have that $f$ is nonnegative.
Now, let's define an $f$ that does not meet such difficulty. Define $f(x)=0.5$ for all $x\in[0,1]$. It meets all the requirements: the range lies in $[0,1]$, multiple composition is well-defined, and the function is positive and continuous.
We have $f(x)=0.5\le0.5$, but $f^n(x)=0.5\gt0.5^n$ for all $n\ge2$.
As you may notice, it is even a counterexample to what you want to prove, because $$\lim\limits_{n\to \infty} 0.5^{\frac{1}{n}}=1\not=0.5.$$
However, I think you may have misunderstood the theorem you want to prove. The correct formula you want to prove might be $$\lim\limits_{n\to \infty} \left[\int^{b}_{a}(f(x))^ndx\right]^{\frac{1}{n}}=\max\{f(x):x\in[a,b]\}.$$ This is a well-known fact about $L^p$ norm of functions. You may want to try to prove this "correct" version of the formula.