I'm looking for a reference, if such a references exists.
So there are currently at least two approaches to topology.
The point-set or "classical" approach to topology, which concerns itself with ordered pairs $(X,\tau)$ called topological spaces.
The "pointless" approach to topology, which concerns itself with (particular kinds of) lattices $(\tau,\wedge,\vee)$ called frames. (For more information, see e.g. Wikipedia.)
I'm interested in a concept halfway between 1 and 2. We might call it "the classical approach, but with lattices."
In particular, rather than studying point-set topological spaces $(X,\tau)$, we concern ourselves with "lattice-theoretic" topological spaces $(P,\tau)$, where $P$ is a lattice that is isomorphic to a powerset lattice, and $\tau$ is a subset of $P$ that is closed with respect to arbitrary joins etc.
The main motivation: We may be able to weaken the requirement that $P$ needs to be isomorphic to a powerset, and thereby obtain a more general theory, which is still classical in flavor.
Has this idea been studied before? If so, a reference recommendation would be great.
I do not have a precise reference. But this may give you something to look at and/or some people to ask.
If you had asked me the similar question about convexity instead of topology, I would've given you a positive answer. The classical point-set topology satisfies the following axioms on the topology of closed sets $\tau \subseteq \mathcal{P}(X)$:
The collection of all convex subsets $C$ on an affine space $E$ satisfy a similar collection of properties:
So one sees that there are certain similarities between convex structures and topological structures.
Now, for convex structures, something like what you proposed has been developed. It is sometimes called "abstract convexity", and one formulation is something like this:
Note that $X$ does not have to be the powerset lattice for a set. It turns out that many facts of convex analysis can be reproduced in this more general context. See, for example, Ivan Singer's Abstract Convex Analysis or MLJ van der Vel's Theory of Convex Structures.
Given the similarity between convex structures and topological ones (for example, the convex hull operator is almost a closure operator in the Kuratowski sense), maybe the sort of "topology as a subset of a complete lattice" point of view can be found linked to from the literature in abstract convexity, and maybe the experts in that area can point you in the right direction so-to-speak.