I have some doubts regarding function notation:
First If I present a function I write:$f(x)$
If I write it's inverse:$f^{-1}(x)$
So why doesn't$f(f(x))=f^2(x)$
Second If $\frac{df(x)}{dx}=f'(x)$ and $\frac{d^2f(x)}{dx^2}=f''(x)$.
So how do you write probably $\frac{d^nf(x)}{dx^n}=f'(x)$.
Does like this:$f'''^{\cdots\text{n times}}(x)$?
Third What difference between $f^n(x),f(x)^n\text{ and }(f(x))^n$
This is indeed ambiguous. Unfortunately $f^n$ is sometimes used to refer to the $n$-th iterate of a function and other times is commonly used to refer to the $n$-th exponentiation. The same notation was adopted by different branches of mathematics to mean different things and both persist in using it so. Thus when ever you see $f^2$, you should check the context to determine if the author meant the composition $f\circ f$ or the product $f\cdot f$.
More confusingly, the convention that $f^{-1}$ means the iterate inverse of a function has become the standard even when $f^n$ is otherwise used for exponentiation.
Some mathematicians adopt the notation $f^{\circ n}$ to be clear that they mean the $n$ iterate.
We never go passed three primes. The convention is that $f^{(n)}$ means the $n$-th derivative of a function (with respect to its argument).
Clarity. $f^n(x)$ is ambiguous; it may mean $f\circ f^{n-1}(x)$ or commonly $f\cdot f^{n-1}$. It is possible that $f(x)^n$ could be parsed as $f(x^n)$. However, $(f(x))^n$ is fairly unambiguous; although it is more chunky.