I noticed:
- Fixing $A_{ij}=1$ would imply $i$-th column and $j$-th row are all $0$'s
- From there, I constructed a few matrices with small $n$ and hypothesized $f(n) = \lfloor{n/2}\rfloor \cdot \lceil{n/2}\rceil$
- Interpreting $A$ as an adjacency matrix: $A^2=0$ implies you can't get from $i$ to $j$ in 2 hops. This reminds me of the max number of edges in a bipartite graph of n nodes; if you consider the edges as directed and the two sets as source/sink sets. I wonder if this relates to max flow.
- Searching "nilpotent binary matrix" got me to Nilpotent binary matrices over finite fields.
- I realized directed edges are not necessary in bipartite graph interpretation -- the problem could be considered as Maximum number of edges in a bipartite graph
I feel there are multiple proof approaches. There is something very classic and familiar going on which I can't put my finger to, and I wonder what non-graph approaches would be especially.
I know that you ask about non-graph approaches in particular. I still felt like a complete proof using graph theory might add something to this thread. I use exactly the technique that you propose in your post.
We want to prove that the maximum number of $1$'s in an $n \times n$ binary matrix $A$ with $A^2 = 0$ is $\lfloor \frac{n}{2} \rfloor \lceil \frac{n}{2} \rceil$.
Let $A$ be an arbitrary matrix of this form. Consider the directed graph $G$ with adjacency matrix $A$. As pointed out in the question, $A^2 = 0$ implies that the longest directed path in $G$ has length at most $1$. Hence no vertex in $G$ has an outgoing and an incoming edge. Therefore there exist only sources, sinks and isolated vertices in $G$. Let $U$ be the set of vertices which are sources and let $V$ be the set of vertices which are sinks or isolated vertices. Clearly all edges in $G$ must go from $U$ to $V$. Hence there are at most $|U||V| = |U| (n - |U|)$ edges in $G$.
The function $f(k) = k(n - k) = nk -k^2$ has a global maximum at $k = \frac{n}{2}$. The two global integer maxima are at $k = \lfloor \frac{n}{2} \rfloor$ and $k = \lceil \frac{n}{2} \rceil$ with value $f(\lfloor \frac{n}{2} \rfloor) = f(\lceil \frac{n}{2} \rceil) = \lceil \frac{n}{2} \rceil \lfloor \frac{n}{2} \rfloor$. Applying this to our argument above we get that there are at most $\lceil \frac{n}{2} \rceil \lfloor \frac{n}{2} \rfloor$ edges in $G$.
Every edge in $G$ corresponds to exactly one $1$-entry in $A$. Hence $A$ has at most $\lceil \frac{n}{2} \rceil \lfloor \frac{n}{2} \rfloor$ entries with a $1$.
We still have to prove that there actually exists a binary $n \times n$ matrix $A$ with $A^2 = 0$ and with exactly $\lceil \frac{n}{2} \rceil \lfloor \frac{n}{2} \rfloor$ $1$-entries. Such a matrix can be constructed as follows:
$$ A_{i, j} = 1 \iff i \leq \frac{n}{2} \; \And \; j > \frac{n}{2}$$