Consider a mixed strategy matrix game where the payoff matrix is (for simplicity)
\begin{bmatrix} & a & b\\ a & u^1_{aa},u^2_{aa} & u^1_{ab},u^2_{ab}\\ b & u^1_{ba},u^2_{ba} & u^1_{bb},u^2_{bb} \end{bmatrix}
Then we create the matrices $A,B$
where,
$ A = \begin{bmatrix} u^1_{aa} & u^1_{ab}\\ u^1_{ba} & u^1_{bb} \end{bmatrix}$ $\quad B = \begin{bmatrix} u^2_{aa} & u^2_{ab}\\ u^2_{ba} & u^2_{bb} \end{bmatrix}$
Let $x, y$ be the mixed strategy for player 1 and 2 respectively. Is the following the correct payoff function for player 1 and player 2?
Player 1 payoff: $x^T Ay$
Player 2 payoff: $y^T B^T x$
If so, I have some questions:
why is $x^TAy$ for player 1? Why is it not $y^TAx$?
why is there a transpose on $B$ for player 2's payoff?
Both are correct. Recall that the payoff is a number so
$$ y^TB^Tx=(y^TB^Tx)^T=x^T(B^T)^T(y^T)^T=x^TBy$$
Personally, I would keep the same notation ($x$ always on the left with $^T$ sign) for both payoffs.