Consider the function:
$h\left(m,n,p\right):=\frac{n-1}{m}\left(\begin{array}{c} n-2\\ n-m-1 \end{array}\right)\cdot\int_{0}^{p}x^{n-m-1}\left(1-x\right)^{m-1}dx$
for $m=1,\ldots,n$, $n\geq2$ and $p\in\left(0,1\right)$. Show that it is unimodal wrt $m$ and determine the term $m^{*}$ that corresponds to the peak of $h\left(\cdot,n,p\right)$.
Here are my steps so far. I consider the difference: \begin{aligned}h\left(m,n,p\right)-h\left(m-1,n,p\right)= & \frac{n-1}{m}\left(\begin{array}{c} n-2\\ n-m-1 \end{array}\right)\cdot\int_{0}^{p}x^{n-m-1}\left(1-x\right)^{m-1}dx\\ & -\frac{n-1}{m-1}\left(\begin{array}{c} n-2\\ n-m \end{array}\right)\cdot\int_{0}^{p}x^{n-m}\left(1-x\right)^{m-2}dx\\ = & \frac{n-1}{m\left(m-1\right)}\left(\begin{array}{c} n-2\\ n-m \end{array}\right)\cdot\int_{0}^{p}x^{n-m-1}\left(1-x\right)^{m-2}\cdot\left\{ n\cdot\left(1-x\right)-m\right\} dx \end{aligned}
The sign of the difference is determined by the term in brackets which is strictly decreasing in $x$ and it is first positive and then negative as $x$ goes from $0$ to $p$. Hence, $h\left(\cdot,n,p\right)$ must have a unique peak at $m^{*}$. But can we specify $m^{*}$ in closed form? It seems that $m^{*}$ is close to and greater than $n\cdot\left(1-p\right)$... How to formally show that?
Here is a proof for my claim. The coefficients $d_{k}=\left(\begin{array}{c} n-1\\ k-1 \end{array}\right)\,p^{k}\,\left(1-p\right)^{n-k}$ correspond to the density of the Binomial distribution with parameters $n-1$ and $p$ with a unique mode at $\left\lfloor n\cdot p\right\rfloor $. Then, $h\left(m,n,p\right)$ corresponds to the running average $\frac{1}{m}\sum_{k=1}^{m}d_{k}$. Let $m^{*}$ be the (unique) maximizer of $h\left(\cdot,n,p\right)$. It is well known that the running average of a monotone sequence follows the same monotonicity. Thus, if $d_{1}>d_{k}$ for $2\leq k\leq n-1$, then $m^{*}=1$. Similarly, if $d_{k}<d_{n-1}$ for $1\leq k\leq n-2$, then $m^{*}=n-1$. However, if $d_{k}$ is interior unimodal, $m^{*}$ is greater or equal to the maximizer of $d_{k}$. Overall, $m^{*}\geq\left\lfloor n\cdot p\right\rfloor $.