In cryptography, non-linearty is an important property in symmetric primitives that ensures trivial linear analysis cannot succeed; associativity is a property that is essential in public-key (a.k.a. asymmetric) cryptography that ensures the correctness key exchange and digital signature schemes.
It's a generally true, that most non-linear operations are also non-associative, but if there exists a associative non-linear operation, it would bridge a major gap between practical and ideal public-key cryptography.
Therefore, I'd like to ask: had there been a known associative binary operation the result of which cannot be represented as a non-trivial linear expression of its input.
e.g.
(One of those) Disallowed trivial linear expression: $x=f(a,b)=a+b+n$.
$x=gcd(a,b)$ can be represented as $x=n \cdot a + m \cdot b$ with $n,m \in Z$, therefore it's excluded.
Let $X$ be any nonempty set, let $x$ be an element of $X$, define a binary operation by $yz=x$ for all $y$ and all $z$ in $X$. There is no group operation here, since $X$ is not a group (if it has more than one element), there is no scalar multiplication here, and no division, so this must be exactly what you want. If it isn't what you want, that's a sign that you haven't thought your problem through, and haven't found a way to express what you really want from this operation.
EDIT: The question currently reads, "Therefore, I'd like to ask: had there been a known associative binary operation the result of which cannot be represented as a non-trivial linear expression of its input" (though I'm sure it will change a few more times before we're done – in particular, I imagine that by "non-trivial" OP actually means "trivial"). So, here goes:
On the set of real numbers exceeding 1, consider the operation given by $a*b=a^{\log b}$. It is a simple exercise in the properties of the logarithm function to prove that this operation is associative. I know of no interpretation of the adjectives under which this operation would be called either trivial or linear.