I am intending to prove the Theorem 2 mentioned in this paper. Judging from the statement of the theorem, it is supposed to be true for both normal-form games and sequential games.
Define
$$R_I^T = \frac{1}{T} \underset{\sigma_i^{'} \in \Sigma_i}{\max} \underset{t=1}{\overset{T}{\sum}}u_i(\sigma_i^{'},\sigma_{-i}^t) - u_i(\bar{\sigma}^T)$$
The theorem states that if $R_i^T < \varepsilon \; \forall i$ in a two player game, i.e:
$$ \frac{1}{T} \max_{\sigma_i'\in\Sigma_i} \sum_{i=1}^T u_i(\sigma_i', \sigma_{-i}^t) - u_i(\overline\sigma^T) < \varepsilon.$$
then $\bar{\sigma}$ is a $2-\varepsilon$ Nash Equilibrium, or:
$$ \max_{\sigma_i'\in\Sigma_i} u_i(\sigma_i', \overline\sigma^T_{-i}) - u_i(\overline\sigma^T) \leq 2\varepsilon, \; \forall i$$
Thanks in advance!
If the equation $R_i^T < \varepsilon \; \forall i = 1,2$ is unrolled, the equation becomes more clear:
\begin{equation} \frac{1}{T}\underset{\sigma_1^{'}}{\max}\underset{t=1}{\overset{T}{\sum}} \sigma_1^{'} U_1 \sigma_2^t - \bar{\sigma_1}U_1\bar{\sigma_2} = \underset{\sigma_1^{'}}{\max}\sigma_1^{'}U_1 \bar{\sigma}_2 - \bar{\sigma_1}U_1\bar{\sigma_2} < \epsilon \tag{1}\label{1} \end{equation}
and
$$ \frac{1}{T}\underset{\sigma_2^{'}}{\max}\underset{t=1}{\overset{T}{\sum}} \sigma_1^{t} U_2 \sigma_2^{'} - \bar{\sigma_1}U_2\bar{\sigma_2} = \underset{\sigma_2^{'}}{\max} \bar{\sigma}_1 U_2 \sigma_2^{'} - \bar{\sigma}_1 U_2 \bar{\sigma}_2 < \epsilon \tag{2}\label{2} $$
where I wrote the expected utilities I wrote in matrix form.
Using the fact that $U_1 = -U_2$, the maximization of \ref{2} can be rearranged as:
$$\bar{\sigma}_1 U_1 \bar{\sigma}_2 < \underset{\sigma_2^{'}}{\min}\bar{\sigma}_1 U_1 \sigma_2^{'} + \epsilon$$
because $\underset{\sigma_2^{'}}{\max}\bar{\sigma}_1 U_2 \sigma_2^{'} = \underset{\sigma_2^{'}}{\max} -\bar{\sigma}_1 U_1 \sigma_2^{'} = -\underset{\sigma_2^{'}}{\min}\bar{\sigma}_1 U_2 \sigma_2^{'}$. Substituting this into the expression in \ref{1} we get:
$$\underset{\sigma_1^{'}}{\max}\sigma_1^{'}U_1 \bar{\sigma}_2 - \epsilon< \bar{\sigma_1}U_1\bar{\sigma_2} < \underset{\sigma_2^{'}}{\min}\bar{\sigma}_1 U_1 \sigma_2^{'} + \epsilon $$
By definition of $\epsilon$-NE, the strategy profile $(\bar{\sigma}_1, \bar{\sigma}_2)$ is a $2 \epsilon$ Nash Equilibrium.