Can we define function concatenation of normal functions $$x\in \mathbb R\\x\to f(x)\in\mathbb R$$ as group operator ?
The identity element would be $f(x) = x$, I suppose
Function inverse can be defined as $f^{-1}(x)$ s.t. $f^{-1}(f(x)) = 0$.
In group setting I suppose our group's inverse element could simply be this function inverse?
These are the two seemingly uncontested properties which fit well.
But... what about associativity? $(f \circ g) \circ h = f \circ (g \circ h)$. Are we sure this will be satisfied?
And what about closure? What families of functions would guarantee this? One example should be polynomials, but I am quite sure that inverse element would often not exist within polynomials...
You only need closure under composition and inverses as composition of functions is always associative.
You may prove this as follows: Let $f:X\to Y$, $g:Y\to Z$, $h:Z\to W$ be functions with $X,Y,Z,W$ non-empty sets. Then, for any $x\in X$, we have
$$((h\circ g)\circ f)(x)=(h\circ g)(f(x))=h(g(f(x)))=h((g\circ f)(x))=(h\circ(g\circ f))(x)$$
and so $(h\circ g)\circ f$ and $h\circ(g\circ f)$ are pointwise equal and thus equal as functions.
Note that this "proof" is really just analysis the nature of composition as we just shift the brackets around; there is no hidden meaning; composition is inherently associative.
An example would the space of all continuous functions from $\mathbb R$ to $\mathbb R$ which are invertible, or in other words bijective. Composition of continuous functions gives you a continuous function and the composition of two bijective functions is bijective.
Another classical example is the group of automorphism of an algebraic structure, e.g. for a vector space $V$ the set of all bijective linear maps $\phi: V\to V$.