I am wondering how to calculate the Nash equilibrium in mixed strategies if the normal form game is NOT represented by a square matrix but a non-square matrix.
I have read several books about how it works for bimatrices (that's the basic example) by solving two equations with two unknown values $p$ and $q$ (the probabilities of both players playing their "first" option).
You can also solve the problem graphically if one of the players just has two options, but this is still not a general applicable method, which I am looking for.
Can someone explain how to do it or maybe recommend good literature where the application of this case is explained?
For illustration, I just give one example of a game and it´s mixed strategies (calculated by https://cgi.csc.liv.ac.uk/~rahul/bimatrix_solver/)
$$\begin{array}{c|c|c|} & \text{X} & \text{Y} & \text{Z} \\ \hline \text{A} & 1 & 0 & 0 \\ \hline \text{B} & 2 & 1 & 0 \\ \hline \text{C} & 1 & 2 & 1 \\ \hline \text{D} & 0 & 1 & 2 \\ \hline \text{E} & 0 & 0 & 1 \\ \hline \end{array}$$
The mixed strategies in the nash eq are $\sigma_1 = (0, 0.5, 0, 0.5, 0)$ for row player and $\sigma_2 = (0.5, 0, 0.5)$ for column player.
For 2 by 2 games, the graphing method you are talking about works in most cases. For other cases, a general method is given in the paper behind the bimatrix solver.
Practically speaking though, you first want to eliminate the strictly dominated strategies, and then look for ways that each player can be indifferent.
So in your game, I can see that $A$ and $E$ are strictly dominated by $\frac{1}{2} B + \frac{1}{2} D$. So now we're left with a 3 by 3 game.
Now suppose P2 plays $xX + yY + (1-x-y)Z$. For P1: $$ u_1 (B) = 2x + y, \quad u_1(C) = 1 + y, \quad u_1 (D) = 2-2x-y. $$ We can then try to test the different cases of mixing. I'll do one case illustratively:
P1 plays a mix between $B$ and $C$. In order to mix between $B$ and $C$ and play $D$ with zero probability, we need $2x + y = 1 + y >2-2x-y$. This implies that $x= 1/2$ and $y>0$. So P2 must be playing $\frac{1}{2}X + yY + (\frac{1}{2} - y)Z$. Now if P1 plays $b B + (1-b)C$, then $$u_2 (X) = 1+b, \quad u_2(Y) = 2-b, \quad u_2 (Z) = 1-b.$$ In order for P2 to be playing $X$ with $1/2$ probability, either:
You then want to test the other cases: mixing between $C$ and $D$, mixing between $B$ and $D$, and mixing between $B, C$ and $D$. You can probably make use of symmetry arguments to save some time.