Consider the bimatrix game given by $$A = \begin{bmatrix} 3 &2 &-2 \\ -1 &3 &0 \\ 1 &-2 &0 \\ \end{bmatrix} $$ $$ B = \begin{bmatrix} -2 &3 &-1 \\ -1 &-2 &2 \\ -1 &-1 &-3 \\ \end{bmatrix} $$
Taking $y^*$ to be the optimal strategy for $P_2$, I get $Ay^{*T} = u_1$ and solving the system gives $u_1 = \frac{1}{16}$ and $y^* = (\frac 5{16}, \frac 18, \frac 9{16})$
Similarly let $x^*$ be the optimal strategy for $P_1$, $B^Tx^{*T} = u_2 $ gives $u_2 = -\frac{15}{14} $ and $x^* = (\frac 1{14}, \frac{5}{14}, \frac{4}{7} ) $
This is indeed an optimal strategy according to sage.
Now consider the bimatrix game $$A = \begin{bmatrix}2&1&2\\ -2&-1&3\\ -3&1&-2\\ \end{bmatrix} $$ $$B = \begin{bmatrix}2&1&-2\\ 1&-1&2\\ 2&1&2\\ \end{bmatrix}$$
Taking $y^*$ as optimal strategy for $P_2$ and taking $Ay^{*T} = u_1$ gives $u_1 = \frac{25}{23}$ and $y* = (-\frac 8{23}, \frac{21}{23}, \frac {10}{23})$ which is absurd because probabilities must be positive.
What am I doing wrong? How to calculate the optimal mixed strategy for such games?
In $2 \times 2$ bimatrix games, I have seen this work every time but it is failing in $3 \times 3$ games. What is the method which can be used for larger bimatrix games also?
What you appear to be doing is trying to find a strategy for each player that makes his or her opponent's payoff independent of the pure strategy chosen by that opponent. While a pair of such strategies will constitute a Nash equilibrium if they exist, it's unfortunately very possible (and very likely, in fact, if the matrices of the game have any more than a few rows and columns) that no such pair of strategies will exist.
This online bimatrix game solver returns $3$ Nash equilibria for your second bimatrix game: \begin{array}{l} 1.& \text{Strategy for player }1:\ (0,1,0)\\ &\text{Strategy for player }2:\ (0,0,1)\\ &\text{Payoff to player }1:\pmatrix{0&1&0}\pmatrix{2&1&2\\ -2&-1&3\\ -3&1&-2}\pmatrix{0\\0\\1}=3\\ &\text{Payoff to player }2:\pmatrix{0&1&0}\pmatrix{2&1&-2\\ 1&-1&2\\ 2&1&2}\pmatrix{0\\0\\1}=2\\ 2.& \text{Strategy for player }1:\ \left(\frac{1}{5},\frac{4}{5},0\right)\\ &\text{Strategy for player }2:\ \left(\frac{1}{5},0,\frac{4}{5}\right)\\ &\text{Payoff to player }1:\pmatrix{\frac{1}{5}&\frac{4}{5}&0}\pmatrix{2&1&2\\ -2&-1&3\\ -3&1&-2}\pmatrix{\frac{1}{5}\\0\\\frac{4}{5}}=2\\ &\text{Payoff to player }2:\pmatrix{\frac{1}{5}&\frac{4}{5}&0}\pmatrix{2&1&-2\\ 1&-1&2\\ 2&1&2}\pmatrix{\frac{1}{5}\\0\\\frac{4}{5}}=\frac{6}{5}\\ 3.& \text{Strategy for player }1:\ (1,0,0)\\ &\text{Strategy for player }2:\ (1,0,0)\\ &\text{Payoff to player }1:\pmatrix{1&0&0}\pmatrix{2&1&2\\ -2&-1&3\\ -3&1&-2}\pmatrix{1\\0\\0}=2\\ &\text{Payoff to player }2:\pmatrix{1&0&0}\pmatrix{2&1&-2\\ 1&-1&2\\ 2&1&2}\pmatrix{1\\0\\0}=2 \end{array}
Note that none of these equilibrium strategies makes the payoff to the opponent of the strategy's user independent of that opponent's strategy.
One sure way of finding a Nash equilibrium for any bimatrix game is the Lemke-Howson algorithm.