In Hartshorne III 12.2, $X\to \text{Spec}\ A$ is a morphism, $\mathcal{F}$ is a coherent sheaf on $X$, flat over $\text{Spec}\ A$, $M$ any $A$ module, then we can construct the sheaf associated to the presheaf $\mathcal{F}\otimes_A M$. But according to the last line on that page, this presheaf should be a sheaf. I don't know why it holds?
To show the presheaf $U\mapsto\mathcal{F}(U)\otimes_AM$ is a sheaf, is equivalent to show
$0\to \mathcal{F}(U)\otimes_AM\to \Pi \mathcal{F}(U_i)\otimes_AM\to\Pi \mathcal{F}(U_i\cap U_j)\otimes_AM$ is exact, where $\{U_i\}$ is arbitrary covering of $U$.
If $M$ is a flat this holds, but what if $M$ is not flat?
Since we are evaluating the cohomology of sheaves on $X$, the notation $\mathscr F\otimes_A M$ must be shorthand for a sheaf on $X$! My guess is that $\mathscr F\otimes_A M := \mathscr F\otimes_{\mathscr O_X} f^\ast\widetilde{M}$ is defined to be the tensor product of $\mathscr F$ with the pullback of $\widetilde M$ along the map $f\colon X\to Y = \operatorname{Spec}(A)$.
Indeed, in Liu's book, applying Prop 1.14(b) on p. 163, we find that for any affine open $U\subseteq X$ we have $$f^\ast\widetilde M\rvert_U \cong \widetilde{\left(M\otimes_A\mathscr O_X(U)\right)}$$ after setting $\mathscr G$ to be the pullback sheaf, and $V = Y$. This implies that $f^\ast\widetilde M(U) = M\otimes_A\mathscr O_X(U)$. Further, combining this with Prop 1.12(b) on the previous page, we find that $$\left(\mathscr F\otimes_{\mathscr O_X} f^\ast\widetilde{M}\right)(U) \cong \mathscr F(U)\otimes_{\mathscr O_X(U)}\left(M\otimes_A\mathscr O_X(U)\right) = \mathscr F(U)\otimes_AM$$ which shows that $\mathscr F\otimes_A M$ (as you are defining it on affine opens) really is the sheaf $\mathscr F\otimes_{\mathscr O_X} f^\ast\widetilde{M}$. The last trick is to realize that this latter sheaf is a sheaf, by definition! I.e. $\mathscr F\otimes_{\mathscr O_X} f^\ast\widetilde{M}$ is the sheafification of the presheaf sending $U\mapsto\mathscr F(U)\otimes_{\mathscr O_X(U)} f^\ast\widetilde{M}(U)$, as Hartshorne mentions at the very beginning of his section II.5. The fact that it plays well with evaluation on affine open sets allows us to proceed with the arguments about Cech cohomology on an affine open covering of $X$.