A, B simultaneously show a number of fingers to each other (both hands implies 0-10), money(£) is awarded as follows:
$ \bullet $ If the number of extended fingers is equal, then no exchange
$ \bullet $ If the number of extended fingers between each player differs by two, the player with the lowest number of fingers gives £2 to the other player.
$ \bullet $ Otherwise the player with the lowest number of extended fingers receives £3 from the other player.
I began with A's payoff matrix with each row/column representing the choice of $\{0,1, ... , 10\}$ resulting in $11$ choices each:
$$ \begin{bmatrix} 0 & 3 & -2 & 3 & 3 & 3 & 3 & 3 & 3 & 3 & 3\\ -3 & 0 & 3 & -2 & 3 & 3 & 3 & 3 & 3 & 3 & 3\\ 2 & -3 & 0 & 3 & -2 & 3 & 3 & 3 & 3 & 3 & 3\\ -3 & 2 & -3 & 0 & 3 & -2 & 3 & 3 & 3 & 3 & 3\\ -3 & -3 & 2 & -3 & 0 & 3 & -2 & 3 & 3 & 3 & 3\\ -3 & -3 & -3 & 2 & -3 & 0 & 3 & -2 & 3 & 3 & 3\\ -3 & -3 & -3 & -3 & 2 & -3 & 0 & 3 & -2 & 3 & 3\\ -3 & -3 & -3 & -3 & -3 & 2 & -3 & 0 & 3 & -2 & 3\\ -3 & -3 & -3 & -3 & -3 & -3 & 2 & -3 & 0 & 3 & -2\\ -3 & -3 & -3 & -3 & -3 & -3 & -3 & 2 & -3 & 0 & 3\\ -3 & -3 & -3 & -3 & -3 & -3 & -3 & -3 & 2 & -3 & 0\\ \end{bmatrix} $$
Luckily, I found that each $\{3, 5, 6, 7, 8, 9, 10\}$ were inferior to $\{0\}$ and $\{4\}$ was inferior to $\{1\}$, leaving simplified matrix with strategies $\{0, 1, 2\}$:
$$ \begin{bmatrix} 0 & 3 & -2\\ -3 & 0 & 3\\ 2 & -3 & 0\\ \end{bmatrix} $$
However, I wasn't able to find a nash equilibrium, and hence requesting to find optimal strategies for each player A, B.
In a mixed-strategy equilibrium, each player must be indifferent among the pure strategies to which she assigns non-zero probability.
In the present case, if all three strategies are assigned non-zero probabilities, this leads to (with the probability for strategy $i$ denoted by $p_i$)
$$ 0p_0+3p_1-2p_2=-3p_0+0p_1+3p_2=2p_0-3p_1+0p_2\;. $$
Together with the normalization constraint $p_0+p_1+p_2=1$, this is a system of three linear equations in three variables. Equating the sum of the first and third expression to the second yields
$$ 2p_0-2p_2=-3p_0+3p_2 $$
and thus $p_0=p_2$. Thus the second expression is $0$. (With hindsight, we could have concluded from the symmetry of the game that the expected value of each strategy is $0$.) Then either equation yields $p_1=\frac23p_0$, and normalization then implies $p_0=p_2=\frac38$ and $p_1=\frac14$. The fact that all three probabilities come out as non-zero shows that the assumption that they are all non-zero was self-consistent.