\begin{align}&\mbox{Is there a closed form for} \\[2mm]&\int_0^{\pi/2} \sin^{2}\left(\, nx\,\right)\sin\left(\, mx\,\right)\cot\left(\, x\,\right) \,{\rm d}x\ \quad\mbox{where}\quad m, n\ \mbox{are positive integers ?.} \end{align}
I tried to turn product of sines into a sum but it didn't really get much easier. WolframAlpha doesn't help either.
I'll be grateful for your ideas how to evaluate it.
You could try writing the sines and cosines in terms of complex exponentials.
Say $x^n - y^n = (x-y)(x^{n-1} + x^{n-2}y + \cdots + x y^{n-2} + y^{n-1} $; hence you can write
$\begin{equation*} \frac{\sin(mx)}{\sin(x)} = \frac{e^{imx}-e^{-imx}}{e^{ix} - e^{-ix}} = e^{i(m-1)x} + e^{i(m-2)x} e^{-ix} + \cdots + e^{ix} e^{-i(m-2)x} + e^{-i(m-1)x} = \sum_{j=0}^{m-1} e^{i(m-1-j)x}e^{-ijx} = \sum_{j=0}^{m-1}e^{i(m-1-2j)x} \end{equation*}$
hence, you have
$\begin{equation*} \sin^2(nx) \sin(mx) \cot(x) = \sin^2(nx) \cos(x) \frac{\sin(mx)}{\sin(x)} = \frac{2-e^{2inx}-e^{-2inx}}{4} \frac{e^{ix} + e^{-ix}}{2} \Big(\sum_{j=0}^{m-1}e^{i(m-1-2j)x} \Big) \end{equation*}$
Just combine the exponentials, and use the fact
$\begin{equation*} \int_0^{\frac{\pi}{2}} e^{i(m-n)x} dx = \begin{cases} \frac{\pi}{2} \text{ if } m = n \\ \frac{(i)^{m-n}-1}{m-n} \text{ if } m \neq n \end{cases} \end{equation*}$