What is the meaning of points $\tau$ in the upper half-plane which are not CM-points but with $j(\tau)$ an algebraic integer?

Question

Due to the relation between imaginary quadratic fields and complex multiplication ($CM$) one knows that for all $CM$-points in the upper half-plane

$$\mathbb{H}:=\{z \in \mathbb{C}:\ Im\ z>0\}$$

the value $j(\tau)$ of the modular $j$-function is an algebraic integer.

Is there an arithmetic significance also of the other points $\tau \in \mathbb{H}$ with $j(\tau)$ an algebraic integer?