tl;dr: I've tried to construct a different way to formalize "topological spaces" than via open sets or neighborhoods. I have not seen this approach but it may have been done before. The definition as it currently stands is not satisfactory (see discussion below).
t-topologies
Consider a pair $(X,t)$ where $t:X\to \mathcal P(\mathcal P(X))$, i.e. $t(x)\subseteq \mathcal P (X)$ (which intuitively gives for every $x$ the set of subsets of $X$ that "x touches") satisfies the following axiom:
Axiom 1. For any $x\in X$ and any $U\subseteq S\subseteq X$, if $U\in t(x)$ then $S\in t(x)$. (intuitively, if $x$ "touches" the set $U$, then it also touches the same set with arbitrary additions to it).
Axiom 2. For any $x\in X$ and any $U,V\subseteq X$, if $U,V\notin t(x)$ then $U\cup V\notin t(x)$ (easily extended to infinite unions). (intuitively, if $x$ doesn't touch either of the sets $U,V$, then it also doesn't touch the combined set).
Call this pair $(X,t)$ a "touch-topology" or "t-topology" (because they're defined via "touch-relations" rather than open sets). Such a pair induces a topology: $(X,T)$, where $U\in T$ iff $\forall x\in U, U^c\notin t(x)$. (Intuitively, a set is open if none of its members touch anything outside the set).
Indeed this is a topology:
the empty set obviously is in $T$.
for any $U_i\in T$, we know that for all $x\in U_i$, $U_i^c\notin t(x)$, therefore since $(\bigcup _j U_j)^c\subseteq U_i^c$ we have $(\bigcup _j U_j)^c \notin t(x)$ by axiom 1. (intuitively, if $x$ touches nothing outside $U_i$, then it also doesn't touch anything outside $\bigcup_j U_j$).
for any $U,V\in T$, we know that for any $x\in U\cap V$ we have $U^c\notin t(x)$ and $V^c \notin t(x)$. Hence by axiom 2, $U^c\cup V^c = (U \cap V)^c\notin t(x)$. Hence $U \cap V\in T$. (easily extended to infinite intersections).
Definition. Let $\mathcal X=(X,t_X)$, $\mathcal Y=(Y,t_Y)$ be t-topologies. A function $f:X\to Y$ is touch-continuous at $x$ if for all $U\in t_X(x)$ it holds that $f(U) \in t_Y(f(y))$. (intuitively, a function is touch-continuous if it doesn't change the things that a point touches).
Theorem. Let $\mathcal X= (X,t_X), \mathcal Y = (Y,t_Y)$ be touch-topologies. If a function $f:X \to Y$ is t-continuous at $x$ then it is also "classically" continuous at $x$ in the topologies $(X,T_X), (Y,T_Y)$ induced by $\mathcal X$ and $\mathcal Y$.
proof. Assume $U\in T_Y$ (an open set). We must show that $f^{-1}(U)\in T_X$. Since $U\in T_Y$, we know that for all $y\in U$, $U^c\notin t_Y(y)$. Hence by t-continuity, $f^{-1}(U^c)\notin t_X(x)$ for all $x\in f^{-1}(U)$. Hence $f^{-1}(U)\in T_X$.
I don't think we can show that continuity implies t-continuity, though I'm not sure.
What's missing from this approach. My intention with this was to represent a topology according to the sets that an object "touches", rather than according to "open sets" (i.e. sets whose elements don't touch the outside).
This approach is similar to the "neighborhood" definition of topology.
However, my approach is not complete, in the sense that we cannot turn any arbitrary topology into a unique t-topology. That is, for any topology, there are multiple t-topologies (non-isomorphic) that induce that topology: for example, consider the topology induced by a directed graph $(X,R)$ ($R$ is a relation on $X$ denoting the arrows of the graph), where the t-topology is: given a node $x$, $U\in t_X(x)$ iff $U$ contains at least one node $y$ such that $Rxy$. It is not hard to show that the t-topology of the transitive closure of any graph induces the same topology as the t-topology of the graph itself.
My questions are:
Has this approach been done before?
Can we add a third axiom so that there is a bijection between topologies and t-topologies? And so that t-continuity and "classical" continuity are equivalent?
are there any problems with this approach?
I think you will be interested in this axiomatization of a topology. This is called a touch relation on a set. You can also read up on the Kuratowski closure operator. A touch relation is basically a Kuratowski closure operator in disguise!
This notion should agree with the notion of limit points of a set in basic analysis and yields the typical metric topology on the reals (that is to say, being a limit point of a subset of the reals is a touch relation on the reals). Thus you also have a nice way to motivate this approach based on what students might know from a first course in analysis.
You can also see that the axiom that differs the most between your approach and touch relations is the handling of transitivity. I think this is why your approach doesn't yield the usual definition of a topology.