In proving the orthogonality of the Legendre polynomials, I have encountered this problem
$$ \int_0^\pi \sin(m\theta) \frac{\partial^n }{\partial \theta^n} [(\cos^{2}\theta -1)^n] d\theta$$
with $m > n$ and $m+n$ an odd integer.
I know that
but then I don't know how to move on, can anyone help me with that? Thank you in advance.

Integrating by parts we get $$ A=\int_0^\pi \sin(m\theta) \frac{d^n }{d \theta^n} [(\cos^{2}\theta -1)^n] d\theta\\ =\sin(m\theta) \frac{d^{n-1} }{d \theta^{n-1}} [(\cos^{2}\theta -1)^n] {\Large{|}}_{0}^{\pi}-m\int_0^\pi \cos(m\theta) \frac{d^{n-1} }{d \theta^{n-1}} [(\cos^{2}\theta -1)^n] d\theta. $$ The quantity $$ \frac{d^{n-k} }{d \theta^{n-k}} [(\cos^{2}\theta -1)^n] {\Large{|}}_{0}^{\pi}=0,\qquad k\in\mathbb{N} $$ since the derivatives $$ \frac{d^{n-k} }{d \theta^{n-k}} [(\cos^{2}\theta -1)^n] ,\qquad k\in\mathbb{N} $$ are proportional to $1-\cos^2\theta$.
Finally we get $$ A=\int_0^\pi \sin^{2n}\theta\frac{d^n }{d \theta^n}\sin(m\theta) d\theta. $$ Now depending whether $n$ and $m$ are odd or even you can calculate this by applying binomial theorem to obtain the integral as a finite sum.