Do we have a criterion for $j$-invariants that gives $CM$ elliptic curves?

Question

It is well-known that every CM elliptic curve has $j$-invariants which are algebraic integers. Is there any criterion to classify all the $j$-invariant that corresponds to $CM$ elliptic curves? For small degrees, such as over $\mathbb{Q}$, we can do it by hand using modular polynomials. Also, we can do this for any algebraic integer with an arbitrarily large degree since the number of modular polynomials of a given degree is finite. but I can't figure out how can we do this in general, especially, without computers.