Let $G=D_{1000}$ (dihedral group) acts transitive on some set $M$ such that for some $x \in M$ $St(x) = \langle \frac{\pi}{10}, S \rangle$, where $S$ is a symmetry. Find 1) $|M|$, 2) kernel of this action, 3) number of stabilizers.
Soulution.
It's easy to see that $St(x) \cong D_{20}$, and since the action is transitive, it has only one orbit. Then $|M| = |G(x)| = \frac{|G|}{St(x)} = \frac{2000}{40} = 50$.
$\ker G = \{g \in G | (\forall y \in M) g(y) =y\} = \cap_{y \in M}St(y) = $ (since M is one orbit all stabilizers are conjugated) $= \cap_{g \in G}(St(x))^g =^? \require{cancel} \cancel{St(x)}$
(it seems to me that $D_{20}$ is a normal subgroup of $D_{1000}$).UPD. $R_1RR_1^{-1} = R, SRS^{-1} = SRS = R^{-1}$, so we should look only at conjugations of symmetries. It's some exercise to show that for even n the conjugacy class of symmetry is all symmetries of the same type (half of the all symmetries). So, $\ker G$ consists of half of all symmetries and $\langle \frac{\pi}{10} \rangle$
Let G acts on its subgroups with conjugations. Then $\#St = |G(St(x))| = \frac{|G|}{|N_{G}(St(x))|} = \require{cancel} \cancel{1}$, where $ N_{G}(St(x)) $ is the normalizer of $St(x)$ in $G$.
$\newcommand{\T}{\mathcal T}$$\newcommand{\Span}[1]{\left\langle #1 \right\rangle}$$\newcommand{\St}{\mathrm{St}}$We can safely forget about $M$. Orbit-stabiliser tells you that you considering the action of $G$ on the (right, say) cosets of the stabiliser $\T = \Span{ \dfrac{2 \pi}{20}, S }$.
The kernel of the action is, as you correctly state, the intersection of all conjugates of $T$. This is the largest normal subgroup contained in $\T$. Now you noted that $\T$ is not normal in $G$, so the kernel will be $K = \Span{\dfrac{2 \pi}{20}}$. The latter is normal, because all subgroups of the cyclic group $\Span{\dfrac{2 \pi}{1000}}$ are characteristic in it.
As to the normaliser of $\T$, let $g = \dfrac{2 \pi}{1000}$. Then if $e = 0, 1$ we have $$ S^{S^{e} g^{i}} = g^{-i} S^{e} S S^{e} g^{i} = S S g^{-i} S g^{i} = S g^{2 i}. $$ Since $\dfrac{2 \pi}{20} = g^{50}$, we have that $S^{e} g^{i}$ normalises $\T$ iff $i$ is a multiple of $25$. Since $g^{25} = \dfrac{2 \pi}{40}$, it follows that the normaliser is $\Span{\dfrac{2 \pi}{40}, S}$, of order $80$.