For a complex number, the standard notation for the magnitude (or modulus) is $$ \def\a{\alpha} \def\abs{\operatorname{abs}} |\a| = \abs(\a) = \sqrt{\a^*\a} $$ For a matrix, the notion of magnitude is better captured by a norm $($e.g.$\:\|A\|_2)\:$ while the pipe-delimited notation customarily denotes $$|A| = \det(A)$$ The few papers that I've read in Quantum Information Theory adopt the $|A|$ notation.
I've also seen $\abs(A)$ used to denote either the elementwise abs() function or else the matrix function $\sqrt{A^2}.\:$ So I was wondering if there was some other unambiguous function or notation, otherwise I'll just stick with $\sqrt{A^\dagger A}$