I am looking for rotation invariant scalar functions $f(x,y): x,y \in R^3$ that are not some scalar function over the dot product (or norm), i.e. $ f \neq g(x\cdot y, \Vert x \Vert, \Vert y \Vert ) $
Do they exist ?
Edit: Edited to clarify that the norm is just another form of the dot product, and the norm being rotation invariant is really just the dot product being invariant.
$$\vert x\times y\vert$$ is invariant under rotations because vector modulus is.
It uses the dot product, but it is not a scalar function of the dot product of its arguments.
$$f(x,y)=\vert x\times y\vert=\sqrt{(x·x)(y·y)-(x·y)^2}\ne g(x·y)$$