The three faces are A,B,and C. We are given three free rolls. In any free roll if we get A or B, the free rolls reset to 3 and three consecutive C means game over. If we get a total of 10 A's the game ends. Every B pays $1*(the total number of A's encountered till that point).
I need to determine the average no. of free rolls and the average payoff.
A key way to simplify this question is to ask "How many B's do I expect to see before I see an A, and before I see three C's consecutively." This is a mini-game whose answer is useful for solving the overall puzzle.
You can subdivide the overall puzzle into phases based on how many A's you've seen so far (A=0, A=1, A=2, etc.). In each phase, you play the game and count the number of B's you see until you either lose or see another A and progress to the next phase. The number of B's you see does not depend on what phase you're in—it neatly factors out.
As shown below, you can calculate this factor $E$ (the expected number of B's before seeing either an A or three consecutive C's) and $P$ (the probability that the mini-game ended with seeing an A rather than seeing three consecutive C's), in which case the expected payout can be expressed as: $$T = E \left[0 + P + 2P^2 + 3P^3 + \ldots + 9P^9\right]$$
And with a similar trick, if $\ell$ is the expected number of dice rolls in a given phase, then the expected number of dice rolls is:
$$L = \ell \left[1 + P + P^2 + P^3 + \ldots + P^9\right]$$
Let $\alpha, \beta, \gamma$ denote the probabilities of rolling an A, B, and C respectively. If you have three throws remaining, then when you roll the die you might get:
Overall, if $E$ denotes the expected number of B's seen, the above cases give the expectation formula:
$$E = (\gamma^3 + \gamma^2\alpha + \gamma \alpha + \alpha)\cdot 0 + (\gamma^2\beta + \gamma\beta + \beta) \cdot (1+E)$$ $$E = (1+\gamma +\gamma^2)\beta + (1+\gamma + \gamma^2)\beta E$$ $$E = \frac{\beta(1+\gamma + \gamma^2)}{1-\beta(1+\gamma+\gamma^2)}$$
We also need the probability that such a mini-game ends with seeing an A, as opposed to seeing three C's consecutively. By a similar formula, this probability is:
$$P = \alpha (1 + \gamma + \gamma^2) + \beta(1 + \gamma + \gamma^2) P + 0 \cdot \gamma^3 $$ $$P = \frac{\alpha (1+\gamma+\gamma^2)}{1-\beta(1+\gamma+\gamma^2)}$$
Now, we can solve the overall expected cost problem.
If you have seen $m$ A's so far, the expected number of B's until you see the next A (or lose the game) is $E$ (as computed above). The expected payout in that case is $mE$. Then, with probability $P$ you continue the game with $m+1$ A's seen so far, otherwise you have seen 3 C's in a row and lost.
When the game ends after seeing 10 A's, the total expected payout $T$ is therefore:
$$T = E \left[P + 2P^2 + 3P^3 + \ldots + 9P^9\right]$$
We can solve the expected length problem (expected number of tosses) using the same approach; if $\ell$ denotes the expected number of dice thrown before you see an A or three consecutive C's, then by my math:
$$\ell = \gamma^3(3) + \gamma^2\alpha(3) + \gamma\alpha(2) + \alpha(1) + \gamma^2\beta(3+\ell) + \gamma\beta(2+\ell) + \beta(1+\ell)$$
$$\ell = 3\gamma^3 + \left[3\gamma^2 + 2\gamma + 1\right](\alpha+\beta) + \left[ \gamma^2 +\gamma + 1 \right]\beta \ell $$
$$\ell = \frac{3\gamma^3 + (3\gamma^2 + 2\gamma + 1)(1-\gamma)}{1-(\gamma^2 +\gamma + 1)\beta}$$ Then we can determine the expected number of dice $L$ thrown during the overall game, using the quantity $P$ computed above:
$$L = \ell (1 + P + P^2 + P^3 + \ldots + P^9)$$