Is there a generalization of topological spaces which works on equivalence classes of subsets?
To be a little bit more precise, I would think of something like the following:
Let $X$ be a set and $P(X)$ its power set. On this power set, we have an equivalence relation which gives rise to the equivalence classes $[A]$, where $A \subset X$. Let us denote by $\mathcal C$ the set of all equivalence classes. I think that we need to assume that the equivalence relation is consistent with taking unions and intersections, i.e., we assume \begin{align} [A \cap B] &= [A] \cap [B] := \{ C \cap D \;\mid\; C \in [A], D\in[B]\}\\ [A \cup B] &= [A] \cup [B] := \{ C \cup D \;\mid\; C \in [A], D\in[B]\} \end{align}
Edit: Some time after posting this question, I have realized that this should boil down to the following: Fix an arbitrary subset $N \subset P(X)$ which is closed under (finite) unions and (finite) intersections. We call the sets in $N$ null sets. These null sets give rise to the equivalence relation $A \sim B$ iff $A \mathbin\Delta B \in N$.
Now, we build a topology $\tau \subset \mathcal C$ and we assume
- $[\emptyset],[X] \in \tau$
- $[A] \cap [B] \in \tau$ for all $[A],[B] \in \tau$
- The assumption concerning infinite unions seems to be more delicate, since it is not clear how this should be defined.
On the functions from $X$ to $\mathbb R$ (or some other image space), we can define an equivalence relation by $f \sim g$ iff $\{ x \in X \;\mid\; f(x) \ne g(x) \} \in [\emptyset]$. Now, it should also be possible to study continuity properties of these functions by requiring that the preimages of open sets are open w.r.t. our generalized topology. Note that these preimages can be understood as an equivalence class in $\mathcal C$.
A precise application I would have in mind would be the following: Let $(0,1)$ be the unit interval equipped with the Lebesgue measure. Two sets are defined to be equivalent, if their symmetric difference is (measurable and) a null set. An equivalence class $[A]$ is said to be open, if for every $\varepsilon > 0$, there is an open set $B$ of measure smaller than $\varepsilon$ such that $A \cup B$ is open. Then, the continuous (equivalence classes of) functions should be precisely the measurable functions.
The problem is that you would need to be precise about how the intersection and union operations interact with your equivalence relation. In particular you need that if $[A]$ and $[B]$ are equivalence classes then $[A]\cap[B]$ is an equivalence class as well, which is definitely not true for general equivalence relations.
For example:
Let $X=\mathbb{R}$ and define an equivalence relation $\sim$ on $2^{\mathbb{R}}$ by defining the following equivalence classes:
$$\mathcal{A}_{1}:=\{\text{ All singletons}\}$$
$$\mathcal{A}_{2}:=2^{\mathbb{R}}\setminus\mathcal{A}_{1}$$
Then note that according to your definition $\mathcal{A}_{1}\cap\mathcal{A}_{1}$ isn't well defined because $\{1\}\cap\{1\}=\{1\}\in\mathcal{A}_{1}$, but $\{1\}\cap\{0\}=\emptyset\in\mathcal{A}_{2}$. Thus $\mathcal{A}_{1}\cap\mathcal{A}_{2}$ isn't one of our two equivalence classes.
You may be able to avoid the issue of coherently defining intersection and union by defining a topology in terms of convergence or in terms of a metric.
However the application you want already exists.
As I said in the comments you can define on a metric on certain equivalence classes of subsets of $[0,1]$ (or $(0,1)$ if your prefer) in a relatively natural way.
Let $[0,1]$ be equipped with the Borel $\sigma$-algebra and the lebesgue measure $\lambda$. Define an equivalence relation $\sim$ on $2^{[0,1]}$ by defining
$$A\sim B\iff \lambda(A\Delta B)=0$$
Where $A\Delta B$ is the symmetric difference of $A$ and $B$. One can check that $\sim$ is indeed an equivalence relation. Denoting the equivalence class of a set $A$ by $[A]$ we have that
$$d([A],[C]):=\lambda(A\Delta B)$$
is a well defined metric on the set of $\sim$ equivalence classes.