If $(X, \mathcal O_X)$ is a scheme, what is $\mathcal O_X(U)$ where $U$ is any open subset (not necessarily affine open)?
Below is what I currently know.
Let $A$ be a ring.
Hartshorne defines the structure sheaf on $\operatorname{Spec}(A)$ as:
For an open subset $U \subset \operatorname{Spec}(A)$, we define $\mathcal O(U)$ to be the set of functions $s: U \to \coprod_{\mathfrak p \in U} A_{\mathfrak p}$, such that:
i.) $s(\mathfrak p) \in A_{\mathfrak p}$
ii.) for each $\mathfrak{p} \in U$, there is a neighborhood $V$ of $\mathfrak p$, contained in $U$, and elements $a, f \in A$ such that for each $\mathfrak q \in V$, we have $f \notin \mathfrak q$ and $s(\mathfrak q) = a/f \in A_{\mathfrak q}$.
Vakil first defines a sheaf of rings on the distinguished base $D(f) \subset \operatorname{Spec}(A)$:
Define $\mathcal O_{\operatorname{Spec} A}(D(f))$ to be the localization of $A$ at the multiplicative $\{g \in A \mid D(f) \subset D(g)\}$.
An exercise shows that $\mathcal O_{\operatorname{Spec} A}(D(f)) \cong A_f$, and that the stalk of at a point $\mathfrak p$ is $A_\mathfrak p$.
Since we have a sheaf on a base, we can extend this to all open $U \subset \operatorname{Spec} A$
Define $\mathcal O_{\operatorname{Spec} A}(U)$ to be the set of all $(s_{\mathfrak p})_{\mathfrak p \in U} \in \prod_{\mathfrak p \in U} A_\mathfrak p$ such that
i.) $s_\mathfrak p \in A_{\mathfrak p}$
ii.) for each $\mathfrak{p} \in U$, there is a neighborhood $D(f)$ of $\mathfrak p$, contained in $U$, and an element $a \in A$ such that for each $\mathfrak q \in D(f)$, we have $s_\mathfrak q = a/f^n \in A_{\mathfrak q}$.
I can see how the two definitions are related: given a function $s \in \mathcal O(U)$ as in Hartshorne, we get $(s(\mathfrak p))_{\mathfrak p \in U} \in \prod_{\mathfrak p \in U} A_\mathfrak p$ as in Vakil. The converse is similar.
Both authors define:
A scheme $(X,\mathcal O_X)$ is a ringed space such that any point of $X$ has an open neighborhood $U$ such that $(U, \mathcal O_X|_U)$ is isomorphic to an affine scheme.
I am confused about functions on a scheme $(X,\mathcal O_X)$.
It seems Hartshorne is explicitly defining what functions on an affine scheme should be in his definition, but not what functions on an arbitrary scheme should be. Why?
Vakil writes: Functions on an open subset $U$ of a locally ringed space have values at each point of $U$. The value at $p\in X$ of such a function lies in $\mathcal O_{X,p}/ \mathfrak m_p$ where $\mathfrak m_p$ is the maximal ideal of $\mathcal O_{X,p}$.
Question: I am confused about functions on a scheme $(X,O_X)$. It seems Hartshorne is explicitly defining what functions on an affine scheme should be in his definition, but not what functions on an arbitrary scheme should be. Why?"
Answer: There are many discussions on this site and the MO site on the structure sheaf of a scheme and the relation with "sheaves of functions", real algebraic varieties etc. You find some information here:
Regular rational functions over real projective lines and projective spaces.
https://mathoverflow.net/questions/377922/building-algebraic-geometry-without-prime-ideals/378961#378961
Question: "Vakil writes: Functions on an open subset $U$ of a locally ringed space have values at each point of $U$. The value at $p∈X$ of such a function lies in $O_{X,p}/m_p$ where $m_p$ is the maximal ideal of $O_{X,p}$."
Answer: Locally if $U:=Spec(A)$, a section $s\in \mathcal{O}_U(U)\cong A$ is an element in the ring $A$, and given any prime ideal $\mathfrak{m}\in A$ you get a canonical "evaluation map"
$$ev: A \rightarrow A_{\mathfrak{m}}/\mathfrak{m}A_{\mathfrak{m}} :=\kappa(\mathfrak{m})$$
and $ev(s)$ is the "value" of the section $s$ at the point $\mathfrak{m}$. This is "unintuitive" since the residue field $\kappa(\mathfrak{m})$ changes with the prime ideal $\mathfrak{m}$, hence $s$ is not a function in the classical sense. There is however a way to construct for any Hilbert-Jacobson ring (scheme) $A$ a ringed topological space $Max(A)$ where you only considers maximal ideals in $A$. Such spaces are "more intuitive". One of the reasons for the introduction of prime ideals for "schemes" is to include nilpotent elements in geometry, and with the above construction of $Max(A)$ you get a locally ringed space with nilpotent elements in the structure sheaf. In the above links you find a discussion of this.
Example: Here:
Canonical bundle and skyscraper sheaf
I give an example to illustrate that a section $s$ of the structure sheaf of $S:=Spec(\mathbb{R}[x])$ does not give a function in the classical sense. This is because the (non-generic) points in $S$ have residue field $k:=\mathbb{R}$ or $K:=\mathbb{C}$. When a point has residue field $k$ you get a well defined function $s:S(k) \rightarrow k$. When the residue field is $K$ you have to make a choice. In the case of $S$ it is possible to make such a consistent choice.
For a real differentiable manifold $M$ and an open subset $U\subseteq M$ if you let $\mathcal{O}(U)$ be the set of differentiable real valued functions $f: U \rightarrow \mathbb{R}$ you get a sheaf of functions $\mathcal{O}$ on $M$ and the pair $(M, \mathcal{O})$ is a locally ringed space. In Hartshorne he defines for any commutative unital ring $A$ a locally ringed space $(X, \mathcal{O}_X)$ where $X:=Spec(A)$ is the set of prime ideals in $A$ with the Zariski topology. A section $s \in \mathcal{O}_X(U)$ is not a function in the classical sense: There is no field $K$ with the property that you may interpret the ring $\mathcal{O}_X(U)$ as a "set of maps" $s: U \rightarrow K$ with multiplication and addition induced from the one on $K$.