In $\mathbb{Z}$, the rules are fairly well established, a few minor quibbles notwithstanding. But in, say, $\mathbb{Z}[\sqrt{7}]$, there are, as far as I can tell, no established rules. What I've seen in some places is that the factorization roughly follows what the canonical factorization would be in $\mathbb{Z}$. This seems inadequate to me whether or not we're in a UFD.
So I propose the following rules:
- First, if applicable, a unit other than 1 (e.g., $-1$, $i$). Negative rational primes (like $-3$ or $-47$) are not used.
- Then, primes/irreducibles in order by norm (lowest to highest), with exponents for any factor present more than once. Where there is a choice between $-a \pm b \theta$ and $a \pm b \theta$, the latter shall be preferred.
- If two or more factors have the same norm, they are sorted by signs thus: $-a - b \theta$, $-a + b \theta$, $a - b \theta$, $a + b \theta$. (In some places I've seen the opposite of this).
If these rules are incomplete, then there is some domain of algebraic integers in which there are numbers with factorizations for which these rules would fail to provide a definitive way of resolving questions of order.
For example, factorize $-405$ in $\mathbb{Z}[\sqrt{-14}]$. This is not a UFD and one of the distinct factorizations is the same as in $\mathbb{Z}$: $(-1) \times 3^4 \times 5$. It can also be factorized as $(-1) \times 3^3 \times (1 - \sqrt{-14})(1 - \sqrt{-14})$ or as $(-1) \times 5 \times (5 - 2\sqrt{-14})(5 + 2\sqrt{-14})$ (these are distinct, but if I'm wrong you'll loudly let me know they're not distinct). I suppose in the former there is no conflict between my proposed rules and a desire to parallel the canonical factorization in $\mathbb{Z}$, but in the latter, my proposed rules dictate 5, with its norm of 25, precede those numbers with a norm of 81, even though they kinda correspond to 3 in the $\mathbb{Z}$ factorization.
Whether or not these rules are complete, I will appreciate comments as to whether you like them on an aesthetic level.
EDIT: By $\theta$, I mean that algebraic number that is being adjoined to $\mathbb{Z}$. So for example, if we're talking about $\mathbb{Z}[\sqrt{7}]$, then $\theta = \sqrt{7}$. Thanks to anon for pointing out an important detail I left out.
No, these rules are incomplete. Consider for example, $\mathbb{Z}[\sqrt{2} + \sqrt{3}]$, a quartic domain in which the integers have the form $a + b\sqrt{2} + c\sqrt{3} + d\frac{\sqrt{2} + \sqrt{3}}{2}$, with $\{a, b, c, d\} \in \mathbb{Z}$. I don't know what the norms are for numbers in this domain, but humor me for just a minute and assume for the sake of argument that $4 + 3\sqrt{2} - 2\sqrt{3} + \frac{\sqrt{2} - \sqrt{3}}{2}$ and $4 + 3\sqrt{2} - 2\sqrt{3} - \frac{\sqrt{2} + \sqrt{3}}{2}$ have the same norm (they probably don't, but humor me). How do you resolve which one goes first? For these rules to be complete, it is necessary to have a full understanding of norm in any arbitrary domain. Also, if $\theta = \sqrt{2} + \sqrt{3}$, the whole thing becomes very awkward before we even think about norms.