Here $k$ and $n$ are fixed positive integers.
I need to find the maximum of $f(x) = E[5\min(m, k) +\frac{kx}2]$ for $x$ between 0 and 10.
Here we have $m \sim Bin(n, 1-\frac x{10})$.
The problem here is that $x$ lies in the probability of the binomial random variable.
We can actually write $f(x)$ as a polynomial of $x$ but I'm wondering if there's a way to find the maximum in terms of a binomial coefficient or probability perhaps.
On
The case $n \le k$ is trivial. In the following, we deal with the case $n > k$.
Result: $f(x)$ achieves its maximum at a unique $p$ (so $x = 10 - 10p$) with $\mathrm{Pr}(\mathrm{Bin}(n-1, p) < k) = \frac{k}{n}$.
Remark: @John Smith said the value of x that maximizes the above satisfies $\mathrm{Pr}(\mathrm{Bin}(n-1, 1 - x/10) < n-k) = 1 - k/n$ ("I might have accidentally flipped one of the signs/variables"). If we replace $x$ by $10-x$ in his equation, it is the same as my result, i.e., $\mathrm{Pr}(\mathrm{Bin}(n-1, x/10) < n-k) = 1 - k/n$ is equivalent to $\mathrm{Pr}(\mathrm{Bin}(n-1, 1 - x/10) < k) = \frac{k}{n}$.
$\phantom{2}$
The details are given as follows.
Let $Y = \min(m, k)$ and $p = 1- \frac{x}{10}$. We have, for $i \in \{0, 1, 2, \cdots, k-1\}$, $$\mathrm{Pr}(Y = i) = \mathrm{Pr}(m = i) = \binom{n}{i} p^i (1-p)^{n-i},$$ and $$\mathrm{Pr}(Y = k) = \mathrm{Pr}(m\ge k) = \sum_{i=k}^n \binom{n}{i} p^i (1-p)^{n-i}.$$ We have \begin{align} \mathbb{E}[Y] &= \sum_{i=0}^{k-1} i \binom{n}{i} p^i (1-p)^{n-i} + k \sum_{i=k}^n \binom{n}{i} p^i (1-p)^{n-i} \\ &= \sum_{i=0}^{k-1} i \binom{n}{i} p^i (1-p)^{n-i} + k \Big[ 1 - \sum_{i=0}^{k-1} \binom{n}{i} p^i (1-p)^{n-i} \Big]\\ &= k - \sum_{i=0}^{k-1} (k-i) \binom{n}{i} p^i (1-p)^{n-i}. \end{align} So, we have $$f(x) = \frac{kx}{2} + 5 \Big[k - \sum_{i=0}^{k-1} (k-i) \binom{n}{i} p^i (1-p)^{n-i}\Big].\tag{1}$$ By using $x = 10 - 10p$, (1) can be written as \begin{align} f(p) = 10k - 5kp - 5 \sum_{i=0}^{k-1} (k-i) \binom{n}{i} p^i (1-p)^{n-i}. \end{align} We have $$f'(p) = -5k - 5\sum_{i=0}^{k-1} (k-i) \binom{n}{i} p^{i-1}(1-p)^{n-1-i} (i - np).$$
To proceed, we need the following result. The proof is given later.
Fact 1: It holds that \begin{align} &n \sum_{i=0}^{k-1} \binom{n-1}{i}p^i(1-p)^{n-1-i}\\ &\quad + \sum_{i=0}^{k-1} (k-i) \binom{n}{i} p^{i-1}(1-p)^{n-1-i}(i - np)= 0. \tag{2} \end{align}
Now, from Fact 1, we have $$f'(p) = -5k + 5n \sum_{i=0}^{k-1} \binom{n-1}{i}p^i(1-p)^{n-1-i}. \tag{3}$$ Clearly, $f'(0) = -5k + 5n > 0$ and $f'(1) = -5k < 0$. Thus, $f(p)$ achieves its maximum at some $p\in (0, 1)$ with $f'(p) = 0$.
From (3), we have $$f'(p) = 0 \quad \Longrightarrow \quad \sum_{i=0}^{k-1} \binom{n-1}{i}p^i(1-p)^{n-1-i} = \frac{k}{n}. \tag{4}$$ In other words, $$f'(p) = 0 \quad \Longrightarrow \quad \mathrm{Pr}(\mathrm{Bin}(n-1, p) < k) = \frac{k}{n}. \tag{5}$$
Note that $\mathrm{Pr}(\mathrm{Bin}(n-1, 0) < k) = 1$ and $\mathrm{Pr}(\mathrm{Bin}(n-1, 1) < k) = 0$. Since $\mathrm{Pr}(\mathrm{Bin}(n-1, p) \le k-1)$ (notice: $k-1 < n-1$) is strictly decreasing of $p$ (see [1]), there exists a unique $p = p_0$ such that $\mathrm{Pr}(\mathrm{Bin}(n-1, p_0) < k) = \frac{k}{n}$. Thus, $f(p)$ achieves its maximum at $p_0$.
Numerical experiment: $n = 8, k=5$; From (4), $p_0$ is a real root of $\frac{3}{8}-15p^7+35p^6-21p^5 = 0$, we get $p_0 \approx 0.5789766668$, $x_0 \approx 4.210233332$ and $f(x_0) = 31.82592210$; We use Matlab binornd to generate random numbers $m$ from binomial distribution and do numerical calculation for $f(x)$.
Reference
[1] Monotonicity of the CDF of a Binomial Distribution
$\phantom{2}$
Proof of Fact 1: (2) is written as $$\sum_{i=0}^{k-1} p^{i-1}(1-p)^{n-1-i}\binom{n}{i} [p(n-i) + (k-i)(i-np)] = 0.$$ Maple tells that it is true. I will try to prove it by hand.
EDIT 2021/01/14
I gave a proof here: Prove the identity $\sum_{i=0}^{k-1} p^{i-1}(1-p)^{n-1-i}\binom{n}{i} [p(n-i) + (k-i)(i-np)] = 0$
Corrected 4/16/2020 -- while in bed, I came to the same conclusion also reached by River Li. He scooped me and posted it first :) and also provided a detailed algebraic proof. Full credits to him! Here I will provide a less algebraic and more intuitive argument.
Not a rigorous, nor full, solution -- but perhaps some ideas that others can improve upon?
Claim: I think the maximum point happens when $P(\color{red}{Bin(n-1,y)} < k) = k/n$ for the case $k < n$. (The case $k \ge n$ is trivial and maximized at $x=10$.)
Disclaimer 1: My "proof" for the claim is not quite rigorous. Disclaimer 2: Even if the claim is true, I don't know how to get from there to the OP's desired answer, because solving $P(m < k) = k/n$ is itself non trivial. Modulo these disclaimers, here's my...
"Proof": I find it easier to think about the problem in terms of $y = 1- \frac{x}{10}$, and the question becomes maximizing
$$g(y) = {f(x) \over 5 } = E [\min(m, k)] + k (1-y) = h(y) + k(1-y)$$
where $m = Bin(n, y)$. As the OP pointed out, $h(y) = E [\min(m, k)]$ is a complicated polynomial in $y$, and it is theoretically possible to differentiate it, then find its root (numerically?) etc. However, that is tedious and (at least for me) does not lead to useful insights.
Instead consider a "richer" sample space where there are $n$ i.i.d. variables, $X_1, \dots, X_n$ each one $\sim Uniform([0,1])$. The $m=Bin(n,y)$ trial can be interpreted as flipping $n$ i.i.d. biased coins, which can be defined in this space ("embedded" into these $X_j$ variables) by interpreting the $j$-th flip to be a success if $X_j < y$.
Now consider an infinitesimal increase $y \to y+dy$. What is the probability that such a $dy$ increase causes $m$ and/or $\min(m,k)$ to increase? If so, by how much?
By ignoring higher order terms, clearly the probability that exactly one $X_j \in [y, y+dy]$ is $n \, dy$, and two or more $X_j \in [y, y+dy]$ is zero. I.e. the probability $m$ increments by one is $n \, dy$, and bigger increments have zero probability.
[Edit begins here] Now, $m$ incrementing does not mean $\min(m,k)$ will increment -- the latter happens only if $m < k$ to begin with. Consider the trinomial process dividing $[0,1]$ into three intervals $[0,y], [y, y+dy], [y+dy, 1]$ and let $A, B, C$ be the number of $X_j$'s in each interval respectively. Then $m$ increments iff $B > 0$, and $\min(m,k)$ increments iff $B > 0 \cap A < k$. Now, ignoring higher order terms:
$$P(B > 0 \cap A < k) = P(B = 1 \cap A < k) = P(B = 1) P( A < k \mid B = 1)$$
Meanwhile, conditioning on $B=1$ means we have a binomial process between $A$ and $C$, with $A+C = n-1$, so $A \sim Bin(n-1, {y \over 1-dy}) = Bin(n-1, y)$. Therefore
$$P(B > 0 \cap A < k) = P(B = 1) P( A < k \mid B = 1) = n\, dy \times P(Bin(n-1,y) < k)$$
The above gives the probability that $\min(m,k)$ increments by one, while bigger increments have zero probability.
Since the increment (if it happens) is exactly one, the probability of the increment happening equals the increase in expected value. I.e.
$$ {d \over dy} h(y) = n \times Prob(Bin(n-1, y) < k)$$
and the maximum happens when:
$$ 0 = {d \over dy} g(y) = n \times Prob(Bin(n-1,y) < k) - k \\ \iff Prob(Bin(n-1,y) < k) = {k \over n} \,\,\,\,\,\,\,\,\,\,\square$$