In the following, I adopt the functorial perspective on schemes. That is, for any ring $R$, the category of $R$-schemes is considered a (full) sub-category of the category $\mathcal{C}$ of all functors from $R$-algebras to sets, affine schemes being those functors isomorphic to $\operatorname{Hom}(A,-)$ for some $R$-algebra $A$. The objects of $\mathcal{C}$ are called $R$-functors for short.
Many definitions and basic results on $R$-schemes naturally generalize to $R$-functors. For example, an open (resp. closed) subfunctor of some $R$-functor $F$ is, by definition, a subfunctor $U$ such that for any $R$-algebra $A$ and for any morphism $\varphi \colon \operatorname{Hom}(A,-) \to F$, the preimage $\varphi^{-1}(U)$ is open (resp. closed) in $\operatorname{Hom}(A,-)$ in the usual sense. It is easy to show that any closed subfunctor $C \subseteq F$ has a unique open "complement" $O \subseteq F$, where complementary means that $C \cap O = \emptyset$ and $C(K) \cup O(K) = F(K)$ for all fields $K$ (in the category of $R$-algebras).
Question 1: Does the converse hold? That is: does any open subfunctor have a (non-unique) closed complement?
Looking at the sub-category of schemes, there is a canonical construction for closed complements as we know that any open subscheme has a unique reduced closed complement. Having the notion of a reduced functor would certainly help a lot for proving the converse true. This leads me to the following question:
Question 2: Is there a meaningful generalization of reduced schemes to $R$-functors?
Let's see an example: Let $F$ be the $R$-functor of all elements of finite multiplicative order, that is, $$ F(A) = \{ x \in A : x^n = 1 \text{ for some } n \in \mathbb{N} \} $$ for all $R$-algebras $A$. We define a subfunctor $O \subseteq F$ which, roughly speaking, consists of all elements of composite order. More formally: $$ O(A) = \{ x \in F(A) : x^p-1 \in A^\times \text{ for all primes } p \}. $$ Using the definition mentioned above, one can easily show that $O$ is open in $F$ (although $O$ is defined by an infinite intersection, the pre-image of $O$ into any affine scheme becomes a finite intersection of open subschemes, which is open again).
For any field $K$, the set $O(K)$ indeed consists of all elements of composite order in $K$. So if there is any closed complement $C$ of $O$ in $F$ then $C(K)$ would be the set of all elements of prime order in $K$ - together with $1 \in K$. I have not found such a complement so far.
Question 3: Does $O$ have a closed complement in $F$?
Update 1 (some thoughts) Both open and closed subfunctors of an $R$-functor $F$ can be regarded as compatible collections of ideals.
More concretely, any open subfunctor of $F$ is equivalent to the assignment of radical ideals $I_{A,a} \subseteq A$ for all $R$-algebras $A$ and all elements $a \in F(A)$ such that for all morphisms $f \colon A \to B$ of $R$-algebras, we have $\sqrt{\langle f(I_{A,a}) \rangle} = I_{B,F(f)(a)}$. Let's call such an assignment an open ideal collection of $F$. The corresponding open subfunctor $O$ of $F$ is given by $O(A) = \{ a \in F(A) : I_{A,a} = A \}$.
Analogously, a closed subfunctor of $F$ is equivalent to the assignment of ideals (not necessarily radicals) $I_{A,a} \subseteq A$ for all $R$-algebras $A$ and all elements $a \in F(A)$ such that for all morphisms $f \colon A \to B$ of $R$-algebras, we have $\langle f(I_{A,a}) \rangle = I_{B,F(f)(a)}$. Let's call such an assignment a closed ideal collection of $F$. The corresponding closed subfunctor $C$ of $F$ is given by $C(A) = \{ a \in F(A) : I_{A,a} = 0 \}$.
From this point of view, we immediately see that closed subfunctors have open complements, as we can canonically transform closed ideal collections into open ideal collections by simply taking radicals.
If all open subfunctors would have closed complements, and if there were such a thing as reduced functors, we should also be able to canonically transform closed ideal collections into open ideal collections. For example, maybe an open collection $(I_{A,a})$ could be transformed into a closed collection by defining $J_{A,a}$ to be those ideal generated by $f(I_{B,b})$ for all morphisms $f \colon B \to A$ with $F(f)(b) = a$?
Concerning reduced functors, I actually think we could define the reduction of $F$ as the intersection of all closed subfunctors $C$ of $F$ such that $C(K) = F(K)$ for all fields $K$. Then, as usual, a functor would be reduced if it coincides with its reduction. Can someone confirm that this definition generalizes reduced schemes?
Update 2 (Reduction of question 1 to a single functor!) My previous thoughts (update 1) can be reformulated in a very neat way.
Let us consider two specific $R$-functors. For any $R$-algebra $A$, let $\mathcal{R}(A)$ be the set of radical ideals of $A$, and let $\mathcal{I}(A)$ be the set of ideals of $A$. For any morphism $f \colon A \to B$ of $R$-algebras, let $\mathcal{R}(f) \colon \mathcal{R}(A) \to \mathcal{R}(B)$ be the map $I \mapsto \sqrt{\langle f(I) \rangle}$, and let $\mathcal{I}(f) \colon \mathcal{I}(A) \to \mathcal{I}(B)$ be the map $I \mapsto \langle f(I) \rangle$. So both $\mathcal{I}$ and $\mathcal{R}$ are $R$-functors. We obviously have a canonical morphism $\phi \colon \mathcal{I} \to \mathcal{R}$ given by $I \mapsto \sqrt{I}$.
My previous considerations show that an open subfunctor of a functor $F$ is actually the same as a morphism $F \to \mathcal{R}$. Similarly, a closed subfunctor of $F$ corresponds to a morphism $F \to \mathcal{I}$. Any closed subfunctor of $F$ given by $F \to \mathcal{I}$ can be transformed to its unique open complement by composition with $\phi$. Conversely, given any open subfunctor of $F$ corresponding to $\alpha \colon F \to \mathcal{R}$, finding a closed complement means finding a morphism $\beta \colon F \to \mathcal{I}$ such that $\alpha = \phi \circ \beta$.
Of course this characterization holds for $F = \mathcal{R}$ in particular. In this way we get an equivalent reformulation to question 1:
Question 1': Does the canonical morphism $\mathcal{I} \to \mathcal{R}$ have a section?
Note that the identity morphism of $\mathcal{R}$ defines an open subfunctor of $\mathcal{R}$, namely $O(A) = \{A\}$ for all $R$-algebras $A$. So we can reformulate question 1 even further so that there is only one functor involved:
Question 1'': Does the open subfunctor $O \subseteq \mathcal{R}$ have a closed complement?
This is really surprising to me: Every open subfunctor of every $R$-functor has closed complements if and only if one specific open subfunctor of one specific functor has closed complements!
However, I doubt that this is the case: If there was a closed complement, there would also be a canonical closed complement (the intersection of all closed complements). So there would be a canonical section of $\phi \colon \mathcal{I} \to \mathcal{R}$. This appears to me as highly unlikely...
Concerning question 1:
In the following, we will disprove the statement of question 1. That is:
Proof: For each $R$-algebra $A$, let $F(A)$ be the set of radical ideals of $A$, and for each morphism $f \colon A \to B$ of $R$-algebras let $F(f) \colon F(A) \to F(B)$ be the map sending $I$ to $\sqrt{\langle f(I) \rangle}$. This clearly defines an $R$-functor.
Let $O \subseteq F$ be the subfunctor defined by $O(A) = \{ A \}$ for all $A$. This subfunctor is open in $F$ which can be seen by using the definition: Let $\varphi \colon \operatorname{Spec}(A) \to F$ be any morphism (remember: $\operatorname{Spec}(A) = \operatorname{Hom}(A,-)$ in this setting), then there is a unique radical ideal $I \in F(A)$ such that $\varphi(f) = F(f)(I)$ for all morphisms $f \colon A \to *$ (Yoneda Lemma). We calculate $$ \varphi^{-1}(O)(T) = \{ f \colon A \to T : \sqrt{\langle f(I) \rangle} \in O(T) \} = \{ f \colon A \to T : \langle f(I) \rangle = T \} \text{ for all } T,$$ so $\varphi^{-1}(O) = \operatorname{Spec}(A)_I $ is open in $\operatorname{Spec}(A)$.
We claim that $O$ has no closed complement in $F$. For that purpose, we may assume without loss of generality that $R$ is a field (any closed complement of $O$ stays a closed complement after base change).
The remainder of this proof is indirect: Suppose there is a closed complement $C \subseteq F$ of $O$. First claim: $C(A) = \{ \sqrt{(0)} \}$ for all $A$.
By definition, we have $C(K) = \{ 0 \}$ for all fields $K$. In particular (since $R$ is a field), we have $C(R) = \{ 0 \}$, and since there is a (unique) morphism $f \colon R \to A$ into any $R$-algebra $A$, we have $\sqrt{(0)} = \sqrt{\langle f(0) \rangle} \in C(A)$. That is, $C(A)$ contains the nil-radical for each $A$.
Let $A$ be any integral domain. Then there is the canonical inclusion $f \colon A \to Q(A)$ into its field of fraction. If $C(A)$ would contain any non-zero radical ideal, it would be sent to $Q(A)$ by $F(f)$ which is impossible. So $C(A) = \{ 0 \}$ for all integral domains.
Let $A$ be any $R$-algebra, and let $I \in C(A)$ be any element. If $I$ is not the nil-radical, there would be a prime ideal $\mathfrak{p} \subset A$ such that $I \not \subseteq \mathfrak{p}$. Then we have a canonical morphism $f \colon A \to A/\mathfrak{p}$ into an integral domain, and $F(f)(I)$ would be non-zero. So we have actually shown that $C(A) = \{ \sqrt{(0)} \}$ for all $R$-algebras $A$.
Since $C \subseteq F$ is closed, we know that $\varphi^{-1}(C) \subseteq \operatorname{Spec}(A)$ must be closed for each morphism $\varphi \colon \operatorname{Spec}(A) \to F$ and each $R$-algebra $A$. Let $A = R[X]$, and let $\varphi$ be given by $f \mapsto F(f)(I)$ for $I = (X) \in F(A)$ (Yoneda Lemma). For any $R$-algebra $T$, we compute $$ \varphi^{-1}(C)(T) = \{ f \colon A \to T : f(X) \text{ is nilpotent} \}.$$ So $\varphi^{-1}(C)$ is isomorphic to the functor $\mathcal{N}$ of nilpotent elements, i.e. $$ \mathcal{N}(A) = \{ x \in A \colon x \text{ is nilpotent} \}, \quad \mathcal{N}(f) = f \quad \text{for all } A, f $$ which is famously known for not being affine (see below). But since $\varphi^{-1}(C)$ is a closed subfunctor of an affine scheme, it has to be affine again. This is the desired contradiction! $\square$
Addendum: Proofs for $\mathcal{N}$ not being affine (i.e. representable) can be found on MSE. I'll copy a nice argument from this answer:
All representable functors preserve limits. In particular, for any representable $R$-functor $F$, the canonical map $F \left( \prod_{n} R[X]/X^n \right) \to \prod_{n} F(R[X]/X^n)$ is bijective. But for $F = \mathcal{N}$ this map is not even surjective, as each $X \in R[X]/X^n$ is nilpotent, but $\prod_n X \in \prod_{n} R[X]/X^n$ is not.
Concerning question 2:
As mentioned before, for each $R$-functor $F$ we can define a closed subfunctor $F_r$ as the intersection of all closed subfunctors $C \subseteq F$ such that $C(K) = F(K)$ for all fields $K$. We may call $F$ reduced if $F = F_r$.
There are some immediate consequences:
In the following, we will show that this definition generalizes the usual definition of reduced schemes (which we denote by $X_{\text{red}}$ for each scheme $X$). That is,
Proof: Let's first consider the special case $X = \operatorname{Spec}(A)$ for some $R$-algebra $A$. Then obviously, we have $$ X_{\text{red}} = \operatorname{Spec}(A/\sqrt{0}) = X_r, $$ since $\sqrt{0}$ is the unique largest ideal of $A$ which is sent to $0$ by all morphisms from $A$ into fields.
Now let $X$ be an arbitrary scheme, covered by open affine subschemes $(O_i)_{i \in I}$. Then $X_{\text{red}}$ is the unique closed subscheme of $X$ with $X_{red} \cap O_i = (O_i)_{\text{red}}$ for all $i$. By the properties we have shown so far, we immediately see that $$ X_{red} \cap O_i = (O_i)_{\text{red}} = (O_i)_r \subseteq X_r \cap O_i \text{ for all } i. $$ Since both $X_r$ and $X_{\text{red}}$ are closed subschemes of $X$, we conclude $X_{\text{red}} \subseteq X_r$. On the other hand, we compute $$ X_{\text{red}}(K) = \bigcup_i (O_i)_{\text{red}}(K) = \bigcup_i (O_i)_r(K) = \bigcup_i O_i(K) = X(K) $$ for all fields $K$, which imples $X_r \subseteq X_{\text{red}}$ by definition of $X_r$. So we actually have $X_{\text{red}} = X_r$ for all schemes $X$. $\square$