So the operation is sum defined by $f+'g=f\circ g$ (composite of functions) and usual scalar multiplication.
First, for $(x+y)+z=x+(y+z)$ property, $(f+'g)+'h=(f \circ g)\circ h\ne f \circ (g\circ h)=(f+'g)+'h$. Is it true?
And $c(f+'g)=c(f\circ g)\ne cf\circ cg=cf+'cg$?
The composition of functions is associative, $f\big(g(h(x))\big) = f(g(x))\circ h(x)$ is the same as $f(x)\circ g(h(x))=f\big(g(h(x))\big)$.
For the second property, you are correct that they aren't always identical, $c\big(f\circ g\big)(x)=cf\big(g(x)\big)$, whereas $c f(x) \circ cg(x)=cf\big(cg(x)\big)$ (Assuming you're working on the space of all functions, providing a simple counterexample would suffice to prove that).
We can thus conclude that it isn't a vector space.