Is there any pre-established operator or function that is zero if any of its operands is zero, otherwise negative if any of its operands is negative, otherwise positive?
The sign is the most important thing here; the magnitude (in the non-zero case) could be anything from the largest absolute value to the product or sum of the absolute values of the operands.
I'm tempted to define such an operator in my work (neural network modelling), but I'd first like to be sure that none already exist.
If there is none, would it be common practice to pick an operator (eg. $\otimes$ or $\star$) and redefine it, or should I stick to function notation?
You could define it as:
$$\left|x_1x_2\cdots x_n\right|\min(x_1,x_2,\dots,x_n)$$
If any are zero, it is zero. Otherwise, it is the same sign as the minimum of the $x_i$, so negative if any $x_i<0$ and positive otherwise.
This has the advantage that it is a continuous function.
I have never heard of such an operator with a name.
For a binary operator, you can write it as:
$$a\star b = \min(a^3b^2,a^2b^3)$$
I wouldn't re-use most operator symbols that are well-known, but you can use $\star$ or $\odot$ or plenty of others that have few common meanings. I'd prefer function notation, personally.
As noted in comments, in response to a question I asked in an earlier version of this comment, $$a\star b = \left|ab\right|\min(\mathrm{sgn}(a),\mathrm{sgn}(b))$$ is a continuous, commutative and associative operation of this sort.