I am not a native english speaker. I learnt about defining and non-defining relative clauses from english grammar books. Grammar books tell me not to use commas in defining relative clauses, so I don't understand why there is a comma preceding "where" in these examples:
From Theorem 4.59 (Sylow Theorems) in Anthony W. Knapp's Basic Algebra, Digital Second Edition:
Let $G$ be a finite group of order $p^mr$, where $p$ is prime and $p$ does not divide $r.$
From Hungerford's Algebra:
Theorem 6.7 (Fundamental Theorem of Arithmetic) Any positive integer $n \gt 1$ may be written uniquely in the form $n = p_1^{t_1}p_2^{t_2} \cdots p_k^{t_k}$, where $p_1 \lt p_2 \lt \cdots \lt p_k$ are primes and $t_i \gt 0$ for all $i.$
I think the clause "where $p_1 \lt p_2 \lt \cdots \lt p_k$ are primes and $t_i \gt 0$ for all $i$" is a defining relative clause, since it gives essential information about the form $n = p_1^{t_1}p_2^{t_2} \cdots p_k^{t_k}.$
I looked up the word "where" in three mathematical textbooks, and in similar situations, they all use commas between formulae and the words "where". Is this a convention just in mathematical writing?
A relative clause (for example, in "The dog that she adores is brown") applies to nouns or noun phrases or pronouns, and contains no standalone sentence. In each case above, since the "where..." clause applies to the entire sentence preceding it and does contain a full sentence, it is actually not a relative clause, and the comma is grammatically correct.
To more clearly see that the "where..." clauses above are actually referring to the entire sentences preceding them, observe that the "where..." clause in the first example is implictly two universal quantifications, while in the second example is implictly two existential quantifications. (And in this example, it is implicitly mixed quantification, which has the additional ambiguity of hanging quantifiers! As Andrew Hwang's above comment suggests, relying on the "where..." clause to implicitly declare quantification is bad writing!)
Finally, as pointed out in John's answer and corroborated in Are "where" and "such that" interchangeable?, without a preceding comma, the word "where" typically technically means "whenever/if".