I have a fixed plane that takes the projection of a 3D image and we need to prove that all the rotations, fixing the plane, is a subgroup of SO(3).
From basic understanding I know that the rotations, fixing the plane means it is just SO(2) and thus I need to prove SO(2) is a sub group of SO(3), but I am not sure how to write it mathematically.
To check that a subset of a group is a subgroup, you need to check that it is closed under composition and inverses. In other words, you must check that if you are given two rotations $g$ and $g'$ fixing your plane $P$, then $gg'$ and $g^{-1}$ also fix $P$.
[Note that this subgroup is indeed an isomorphic copy of SO(2) sitting inside of SO(3), but you have to be careful in your phrasing, since unless that plane is the $xy$ plane it won't be the 'standard' copy of SO(2) sitting in SO(3).]