I am taking an introductory course on game theory, which aims at promoting interests on mathematics rather than giving a rigorous treatment. I am not sure how to solve the following problem about mixed Nash equilibrium:
Suppose we have a $3$ -players game where players point up $1$ or $2$ fingers, if only one player points up $1$ finger, he gets $1$ dollar. If only one player points up $2$ fingers, he gets $2$ dollars. Otherwise, they get none.
My work: since the game is symmetrical, it means that all three players would choose the same strategy. Let $x$ be the percentage that the players should point up $1$ finger, my idea is that the optimal percentage should occur when one's expected value is at maximum,
$$E(x)=2(1-x)x^2 + x(1-x)^2=x-x^3$$
From there I conclude that $x=\frac 1{\sqrt 3} \approx 0.577 $
However the answer given is $0.41$ for $1$ finger, $0.59$ for $2$ fingers.
I suspect the problem is that the equilibrium is not at the maximum expected value, since given the mixed strategy calculated using my method, any player would like to increase the percentage for $2$ fingers in order to gain more. So I actually need to find the percentage at which if anyone changes his mixed strategy, he will gain less. I don't have any idea how to find it.
Since the game is symmetric, the three players pick the same strategy. Let $x$ be the probability that a player points up $1$ finger. In order to play a randomised strategy, the player must be indifferent between playing $1$ or $2$. (If he were not indifferent, he would rather play the pure strategy that maximises his utility.) Since $U(1) = (1-x)^2$ and $U(2) = 2x^2$, we obtain $$(1-x)^2=2x^2$$ from which we conclude $x = \sqrt{2}-1 \approx 0.41$.