According to MathWorld, a compact Lie group is a group whose parameters vary over a closed interval. I'm not sure if this definition is rigorous enough.
I've also seen a similar definition here:
Note that the parameter labelling the rotations varies in a compact interval (the interval $[0, 2π)$ in this case). Groups with parameters varying over compact intervals are called compact groups.
Anyway, wouldn't this definition imply that rotations do not form a compact group, since $[0,2\pi)$ is not closed, so it's not compact? Am I missing something?
This is a bad definition. There is indeed a map, even a continuous bijection, from $[0, 2\pi)$ to $SO(2)$. However, this map is not a homeomorphism, so its existence does not imply that $[0, 2\pi)$ and $SO(2)$ have the same topological properties, such as compactness.
In more detail, there is a sequence $x_n = 2 \pi - \frac{1}{n}$ in $[0, 2\pi)$ that does not have a limit, so $[0, 2\pi)$ is not sequentially compact. However, the image of this sequence in $SO(2)$ just converges to the identity, so this doesn't prevent $SO(2)$ itself from being sequentially compact.