Probably it is a simple question but I was not able to find the answer somewhere.
Assume that $\hat{F}_{X}(\cdot)$ is the empirical cdf estimator that refers to the continuous random variable $X$. We know that the variance of this estimate at a point, say $x_{1}$, is $Var(\hat{F}_{X}(x_{1}))=\frac{F_{X}(x_{1})(1-F_{X}(x_{1}))}{n}$. The expression is similar for another different point, say $x_{2}$.
What is the covariance: $Cov(\hat{F}_{X}(x_{1}),\hat{F}_{X}(x_{2}))$ ??
Perhaps it's a bit late, but here's help at a solution.
Recall that $\mathbb{E}[\hat{F}_n(x)] = F(x)$. If we expand out the expression for covariance, we are trying to compute
\begin{align*} \mathbb{E}\left[ \left(\hat{F}_n(x_1) - F(x_1)\right)\left(\hat{F}_n(x_2) - F(x_2)\right)\right] = \mathbb{E}\left[ \hat{F}_n(x_1)\hat{F}_n(x_2)\right] - F(x)F(y). \end{align*}
Now it's easiest to expand out the first term and get \begin{align*} \mathbb{E}\left[ \hat{F}_n(x_1)\hat{F}_n(x_2)\right] = \frac{1}{n^2}\mathbb{E}\left[ \sum_i \mathbb{I}(X_i \leq x_1) \sum_j \mathbb{I}(X_j \leq x_2)\right]. \end{align*}
If $i \neq j$, these are independent and we just get $F(x_1)F(x_2)$. If $i = j$, then we have to think about how these two events in the indicators actually relate. I'll leave it to the reader to get a pretty final formula.