I hope I got the wording for the title right, but I basically mean a smooth function on real numbers or matrices such that $f(x,y)=f(y,x)$ and $f(x,f(y,z))=f(f(x,y),z)$ where the function is not of the form $f(x,y)=g(h(x)+h(y))$.
Does such a function exist?
(But not something like $\max(x,y)$ which is like the limiting function of $(x^N+y^N)^\frac{1}{N}$)
How about $f(x,y)=\max\{x,y\}$? Clearly $\max\{x,y\}=\max\{y,x\}$ and $$\max\{x,\max\{y,z\}\}=\max\{\max\{x,y\},z\},$$ and no $g$ and $h$ as you describe exist. It isn't smooth along the diagonal, however.