Here $R$ is a commutative ring with unity and $\mathrm{Sch}/R$ is the category of schemes over $\operatorname{Spec}R$.
I know that the contravariant functor $\mathcal O$ of open subsets of a scheme over $R$ is not representable in the case of $R=k$ with $k$ field.
If not, let $X$ be a scheme over $k$ and $U\subset X$ such that for each $k$-scheme $S$ and each $V\subset S$ open subset there exists a unique $f:S\to X$ s.t. $f^{-1}(U)=V$. Then, we can imply that there is a unique morphism $f:S\to U$ for each scheme $S$ over $k$. Then $U\cong \operatorname{Spec}k$ because it is the terminal object of the category. Hence $U$ is open by hypothesis, but it also closed since it is a $k$-rational point of $X$. Hence each open set of each scheme $S$ is also closed and this is a contraddiction.
In the case of $R$ commutative ring I can say that $U\cong \operatorname{Spec}R$ as above, but I can not say more.
Any hint or ideas? Thank you in advance
Correct me if I'm wrong but I generally don't think this functor is representable for schemes over $\mathbf{Spec}(R)$:
Let $S$ be such a scheme, not necessarily reduced and $S^{\text{red}}$ its reduced scheme. Then under this functor $\mathbf{O}$, $\mathbf{O}(S)$ and $\mathbf{O}(S^{\text{red}})$ are send to the same set of open subset, because both spaces are the same as topological spaces (but not as schemes).
However, $Hom(S,X)$ and $Hom(S^{\text{red}},X)$ are different, because for e.g. $S=\mathbf{Spec}(A)$ we have that these $Homs$ are the same as $Hom_R(\Gamma(\mathcal{O}_X),A)$ and $Hom_R(\Gamma(\mathcal{O}_X),A/Nil(A))$ where $Nil(A)\subset A$ is the nilradical in $A$.