Let $(X,\mathcal{O}_X)$ be a scheme, and define a presheaf of rings, $\mathscr{F},$ by $U\mapsto\mathcal{O}_X(U)_{\text{red}}$, where for any commutative ring $A$, $A_{\text{red}}$ is the quotient of $A$ by its nilradical.
Is $\mathscr{F}$ a sheaf? In my initial attempts to prove this fact, I arrive at a point where I require the quasi-compactness of $X$. Is $\mathscr{F}$ a sheaf when $X$ is quasi-compact?
Some (optional) background first: Let $(X,\mathcal{O}_X)$ be any (locally) ringed space. The sheaf associated to the presheaf $F$ you describe is denoted by $(\mathcal{O}_X)_{red}$. Then $X_{red} := (X,(\mathcal{O}_X)_{red})$ is the terminal reduced (locally) ringed space equipped with a morphism $X_{red} \to X$. It is known that if $X$ is a scheme, then the same is true for $X_{red}$. In fact, we have $\mathrm{Spec}(A)_{red}=\mathrm{Spec}(A_{red})$ since both sides satisfy the same universal property. Your question is now if we the canonical homomorphism $\mathcal{O}_X(U)_{red} \to (\mathcal{O}_X)_{red}(U)$ is an isomorphism. As remarked above, this is true when $U$ is an affine scheme.
The map $F(U) \to \prod_i F(U_i)$ is injective for every finite open cover $U = \cup_i U_i$. In fact, if $[s] \in F(U)$ lies in the kernel, represented by $s \in \mathcal{O}_X(U)$, this means that each $s|_{U_i}$ is nilpotent, say $s|_{U_i}^{n_i}=0$. It follows $s^n = 0$ with $n := \max_i n_i$, so that $[s]=0$. In particular, the map is injective for every open cover when $U$ is quasi-compact. In particular $F$ is separated when $X$ is noetherian.
Here is an example which shows that $F$ doesn't have to be separated in general: Let $X = \coprod_{n \geq 1} \mathrm{Spec}(\mathbb{Z}/2^n \mathbb{Z})$. Then $\mathcal{O}_X(X)=\prod_{n \geq 1} \mathbb{Z}/2^n \mathbb{Z}$. Here, $2$ is not nilpotent, but we have $2^n=0$ on $\mathrm{Spec}(\mathbb{Z}/2^n \mathbb{Z})$. There are also examples of irreducible schemes, but they are more complicated.
What about the other sheaf axiom? Let $U = \cup_i U_i$ be a finite open cover and let $[s_i] \in F(U_i)$ be compatible elements, i.e. $s_i \in \mathcal{O}_X(U_i)$ such that $s_i|_{U_i \cap U_j}-s_j|_{U_i \cap U_j}$ is nilpotent. We ask if there is a section $s \in \mathcal{O}_X(U)$ such that $s|_{U_i}-s_i$ is nilpotent. I doubt that this is the case, even for two open subsets $U = U_1 \cup U_2$. For example, it may happen that $U_1 \cap U_2$ has a nontrivial nilpotent section which lifts to a section $s_1$ on $U_1$, but $U_1$ and $U_2$ have only trivial nilpotent sections, and then $s_2=0$ gives a counterexample. I will add a specific example when it comes to me ...