Let $x$ be the mixed strategy of player $1$
Then the mixed strategy for player $1$ is calculated with respect to $[1, 0], [0, 1]$, the pure strategies of player $2$.
i.e. $x^*$ = $\max \min x^TA([1,0]^T,[0,1]^T$)
Let $y$ be the mixed strategy of player $2$
Then the mixed strategy for player $2$ is calculated with respect to $[1, 0], [0, 1]$, the pure strategies of player $1$.
i.e. $y^*$ = $\min \max ([1,0],[0,1])Ay$
Why do players compute their mixed strategies with respect to the other player's pure strategies instead the other player's mixed strategies?
This is because the mixed strategies form a convex set and the quantity being optimized by the opponent in the inner optimization is a linear function, which takes its extreme values on the extreme points of the convex set, i.e. the pure strategies.
You may be thinking that this isn't the case because it's not the case for the outer optimization – but note that the outer optimization doesn't optimize a linear function.