Use De Moivre's Theorem to express $\cos{5\theta}, \sin{5\theta}$ in powers of $\sin{\theta}$ and $\cos{\theta}$.
Show that $$\frac{\cos{5\theta}}{\cos{\theta}}=16\sin^4{\theta}-12\sin^2{\theta}+1$$
Find the solutions of $$16\sin^4{\theta}-12\sin^2{\theta}+1=0,\ for\ 0\le\theta\le\frac{\pi}{2}$$
Use this result to find $\sin{\frac{\pi}{10}}$ in the form $a+b\sqrt{k}$, and give the values of $a,\ b\ and\ k$.
For $\cos{5\theta}$, I have $\cos5{\theta}=\cos^5{\theta}-10\cos^2{\theta}\sin^3{\theta}+5\cos{\theta}\sin^4{\theta}$
and for $\sin{5\theta}$, I have $\sin{5\theta}=5\cos^4{\theta}\sin{\theta}-10\cos^2{\theta}\sin^3{\theta}+\sin^5{\theta}$
Then it's easy to get $\frac{\cos{5\theta}}{\cos{\theta}}=16\sin^4{\theta}-12\sin^2{\theta}+1$.
So on, $\frac{\cos{5\theta}}{\cos{\theta}}=16\sin^4{\theta}-12\sin^2{\theta}+1=0$
Since $\cos{\theta}\ne0$, so $\cos{5\theta}=0$
Then $$5\theta=\frac{\pi}{2},\frac{3\pi}{2},\frac{5\pi}{2},\frac{7\pi}{2},\frac{9\pi}{2},$$
$$\therefore \theta=\frac{\pi}{10},\frac{3\pi}{10},\frac{\pi}{2},\frac{7\pi}{10},\frac{9\pi}{10}$$
Since $\cos{\theta}\ne0, \therefore \theta\ne\frac{\pi}{2}, and\ also\ 0\le\theta\le\frac{\pi}{2}$
$$\theta=\frac{\pi}{10}, \frac{3\pi}{10}$$
Then I used the quadratic formula to get $$\sin^2\frac{\pi}{10}=\frac{3\pm\sqrt{5}}{8}$$
But this is wrong somehow. Can anyone give me some hints to finish this question?

You got a correct expression, but it's not what the question asked for. $$\sin^2 \frac{\pi}{10} = \frac{3-\sqrt{5}}{8} \tag{1}$$ is correct (we discard the other root $\frac{3+\sqrt{5}}{8}$ because it is greater than $\sin^2 \frac{\pi}{4} = \frac{1}{2}$). But the question asked for $\sin \frac{\pi}{10}$ in the form $a + b \sqrt{k}$.
To this end, note that $$3-\sqrt{5} = \frac{6-2\sqrt{5}}{2} = \frac{(\sqrt{5})^2 - 2\sqrt{5} + 1}{2} = \frac{(\sqrt{5} - 1)^2}{2}. \tag{2}$$ Hence $$\sin \frac{\pi}{10} = \sqrt{\frac{3 - \sqrt{5}}{8}} = \sqrt{\frac{(\sqrt{5}-1)^2}{16}} = \frac{\sqrt{5} - 1}{4}. \tag{3}$$ This yields $a = -1/4$, $b = 1/4$, $k = 5$.