I am having trouble understanding the jump from the 2nd to 3rd line in equation (36). Somehow the superscripts become j and are being subtracted? The full paper is here (appendix c): https://arxiv.org/pdf/2002.03478.pdf
Proof:
The standard importance sampling (IS) estimator is given by $$ \hat{v}_{I S}^{\pi_{e}}=\frac{1}{N} \sum_{n=1}^{N} w_{0: T}^{(n)} g_{T}^{(n)} $$
where the summation is over all $N$ trajectories in the dataset, and the importance sampling weight $w_{0: t}$ is given by $$ w_{0: t}^{(n)}=\prod_{t^{\prime}=0}^{t} \frac{\pi_{e}\left(a_{t^{\prime}}^{(n)} \mid s_{t^{\prime}}^{(n)}\right)}{\pi_{b}\left(a_{t^{\prime}}^{(n)} \mid s_{t^{\prime}}^{(n)}\right)} $$ The total influence of trajectory $j$ is then: Equation (36): $$ \begin{aligned} I_{j} &=\hat{v}_{-j}-\hat{v} \\ &=\frac{1}{N-1} \sum_{n \neq j} w_{0: T}^{(n)} g_{T}^{(n)}-\frac{1}{N} \sum_{n=1}^{N} w_{0: T}^{(n)} g_{T}^{(n)} \\ &=\sum_{n=1}^{N} w_{0: T}^{(n)} g_{T}^{(n)}\left(\frac{1}{N-1}-\frac{1}{N}\right)-\frac{1}{N-1} w_{0: T}^{(j)} g_{T}^{(j)} \\ &=\frac{1}{N(N-1)} \sum_{n=1}^{N} w_{0: T}^{(n)} g_{T}^{(n)}-\frac{1}{N-1} w_{0: T}^{(j)} g_{T}^{(j)} \\ &=\frac{1}{N-1}\left(\hat{v}_{I S}-w_{0: T}^{(j)} g_{T}^{(j)}\right) \end{aligned} $$ This relation is nothing more then the fact that removing the $j^{t h}$ sample from an average over $N$ samples, $\bar{x}=$ $\frac{1}{N} \sum x^{(n)},$ changes the average by $\frac{1}{N-1}\left(\bar{x}-x^{(j)}\right)$