You pick a nonzero number ($n$) of dice. The dice are tossed simultaneously until one, and only one, is a six. On success there is a payout of $p$ dollars. Otherwise, for each toss of the dice, there is a cost of $c$ dollars. Choose $n$ to maximise your profit.
Myself, I assumed that a maximum of $k$ rounds were played, where $k= p \div c $ . Then, the expected probability of success is: $$1 - (\frac{5n} {6^n})^k$$
However, I found that this doesn’t quite work because the probability of the game reaching the $k$th is itself variable.
I wrote a Python program to simulate the game here,
import random
payout = 300
rcost = 100
# Adjust these as per requirement
nsim = 10000
def compute(ndie):
profit = 0
while True:
c = [random.randint(1, 6) for _ in range(ndie)]
if c.count(6) == 1:
profit += payout
break
else:
profit -= rcost
return profit
x = int(input("Number of die? "))
s = 0
for i in range(nsim):
l = compute(x)
s += l
print("You win {} on average ".format(s/nsim))
Problem credits: Unihedron, source.
The probability of success on a single throw is $\dfrac{n 5^{n-1}}{6^n}$ making the expected number of throws until success $\dfrac{6^n}{n 5^{n-1}}$ and the expected profit from a game played until success $p-\dfrac{c6^n}{n 5^{n-1}}$.
This will be maximised when $\dfrac{6^n}{n 5^{n-1}}$ is minimised, which for integer $n$ is when $n=5$ or $6$.
In those cases, the probability of success on a single throw is $(\frac56)^5$, the expected number of throws until success $(\frac65)^5$ and the expected profit from a game played until success $p-c(\frac65)^5$.
The optimal number of dice does not depend on $p$ or $c$, providing $c \gt 0$.