Question:Let $n \in \mathbb{Z}^{+}$, let $r$ and $s$ be the usual generators of $D_{2n}$ and $\theta=\frac{2\pi}{n}$ .Prove that the map $\varphi : D_{2n}\to GL_2(\mathbb{R})$ defined on generators $\varphi(r) = \left[ {\cos \theta \atop \sin \theta} {-\sin \theta \atop \cos \theta} \right]$ and $\varphi(s) = \left[ {0 \atop 1} {1 \atop 0} \right]$ extends to a homomorphism of $D_{2n}$ into $GL_2(\mathbb{R})$.
Now every element of $D_{2n}$ can be written uniquely as $s^kr^i$ where $0\leq i \lt n$ and $0 \leq k \lt 2$. so now define $\varphi$ as $\varphi (s^kr^i)=\varphi(s)^k \varphi(r)^i $. now we have to show that $\varphi(r)$ and $\varphi(s)$ satisfy the same relation as $D_{2n}$.
Now $\varphi(r)=\begin{bmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \\ \end{bmatrix}$ claim: $\varphi(r^k)=\begin{bmatrix} \cos k\theta & -\sin k\theta \\ \sin k\theta & \cos k\theta \\ \end{bmatrix}$. we will do this by induction.the base case is trivial . let this holds for some $k$ so now
$\begin{align*} &\varphi(r^{k+1}) = \varphi(r)^k \cdot\varphi\\ =&\ \left[ {\cos k\theta \atop \sin k\theta} {-\sin k\theta \atop \cos k\theta} \right] \cdot \left[ {\cos \theta \atop \sin \theta} {-\sin \theta \atop \cos \theta} \right]\\ =&\ \left[ {\cos \theta \cos k\theta - \sin \theta \sin k \theta \atop \cos \theta \sin k\theta + \sin \theta \cos k\theta} {-(\sin \theta \cos k\theta + \cos \theta \sin k\theta) \atop \cos \theta \cos k\theta - \sin \theta \sin k\theta} \right]\\ =&\ \left[ {\cos (k+1)\theta \atop \sin (k+1)\theta} {-\sin (k+1)\theta \atop \cos (k+1)\theta} \right] \end{align*}$.
so we have $\varphi(r^k)=\begin{bmatrix} \cos k\theta & -\sin k\theta \\ \sin k\theta & \cos k\theta \\ \end{bmatrix}$. now for $k=n$ and $\theta=\frac{2\pi}{n}$ we have $\varphi(r^n)=\begin{bmatrix} \cos 2\pi & -\sin 2\pi \\ \sin 2\pi & \cos 2\pi \\ \end{bmatrix}=\begin{bmatrix} 1 & 0\\ 0 & 1\\ \end{bmatrix}=I_2$ and for $\varphi(s^2)= \begin{bmatrix} 0 & 1\\ 1 & 0\\ \end{bmatrix} \begin{bmatrix} 0 & 1\\ 1 & 0\\ \end{bmatrix}=\begin{bmatrix} 1 & 0\\ 0 & 1\\ \end{bmatrix}=I_2$ and $\varphi(s) \varphi(r)=\begin{bmatrix} 0 & 1\\ 1 & 0\\ \end{bmatrix} \begin{bmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \\ \end{bmatrix}= \begin{bmatrix} \sin \theta & \cos \theta \\ \cos \theta & -\sin \theta \\ \end{bmatrix}$ $\implies$ $(\varphi(s) \varphi(r))^2=I_2 \implies \varphi(r) \varphi(s)=\varphi(s) \varphi(r^{-1})$.
now the last thing we have to show if $s^ar^b$ and $s^cr^d$ are two elements of $ D_{2n}$ then $\varphi(s^ar^bs^cr^d)= \varphi(s^ar^b) \varphi(s^cr^d)$.
for $c=0$, $\varphi(s^ar^bs^cr^d)=\varphi(s^ar^{b+d})=\varphi(s)^a \varphi(r)^{b+d}=\varphi(s^ar^b) \varphi(s^cr^d)$
and for $c=1 $, $\varphi(s^ar^bs^cr^d) =\varphi(s^ar^bs^1r^d)= \varphi(s^{a+1} r^{d-b}) = \varphi(s)^{a+1} \varphi(r)^{d-b} = \varphi(s)^a \{\varphi(s)^1 \varphi(r)^{-b}\} \varphi(r)^d =\varphi(s)^a \{(\varphi(r)^{-b})^{-1} \varphi(s)^1\} \varphi(r)^d=\varphi(s)^a \varphi(r)^{b} \varphi(s)^1 \varphi(r)^d= \varphi(s^ar^{b}) \varphi(s^1r^d)$.
I would suggest using von Dyck's theorem.
Since $D_{2n}$ has defining relations $$ r^{n}= s^{2} = (r s)^{2} = 1, $$ it suffices to show that $\varphi(r)$ and $\varphi(s)$ satisfy the same relations.
For instance $$ \varphi(r)^{n} = \begin{bmatrix} \cos( \theta) & -\sin (\theta) \\ \sin (\theta) & \cos (\theta) \end{bmatrix}^{n} = \begin{bmatrix} \cos( n \theta) & -\sin (n \theta) \\ \sin (n \theta) & \cos (n \theta) \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}. $$