Let $O(2)$ denote orthogonal group of set of all linear transformation in planeand $SO(2)$ is a subgroup orthogonal group with determinant 1.Also $SO(2)$ is of index 2 in $O(2)$
Now, my professor says $SO(2)$ consists only of rotation by angle $\theta$ counter clockwise.While the other coset of $SO(2)$ consists only of reflection.
But rotation by $\theta$ clockwise also belong to O(2) . Since it's determinant is -1 , i should be in $O(2)-SO(2)$ but it is not true.
What am i missing? Please explain me where is rotation in clockwise direction? Is it in $SO(2)$ or in $O(2)-SO(2)$
Rotation by $\theta$ clockwise also has determinant $+1$, and it is in $SO(2)$. You should write down the matrix and compute its determinant to see for yourself.
Saying $SO(2)$ consists "only" of counterclockwise rotations is perhaps misleading. We should rather say that every element of $SO(2)$ can be represented as a counterclockwise rotation, but there might be other ways to represent it as well.
It might also help to realize that clockwise rotation by an angle $\theta$ radians is the same as counterclockwise rotation by $2\pi-\theta$.