Let $X = \operatorname{Spec} A$ be a noetherian affine scheme, $U \subseteq X$ open, and $\mathscr{F}$ a coherent sheaf on $U$. Furthermore assume $\mathscr{G}$ is quasi-coherent on $X$ so that $\mathscr{F} \subseteq \mathscr{G}|_{U}$ is a subsheaf. We then want to show that there is a coherent subsheaf $\mathscr{F}' \subseteq \mathscr{G}$ so that $\mathscr{F}'|_{U} \cong \mathscr{F}$. Let $i: U \to X$ be the inclusion morphism.
Hartshorne's hint is to use the canonical $\rho: \mathscr{G} \to i_*(\mathscr{G}|_U)$ which comes from adjunction since $\mathscr{G}|_U = i^*\mathscr{G}$ by definition. This morphism makes sense(is canonical) as a morphism of sheaves on $X$. Hartshorne then wants you to then consider $\rho^{-1} i_* \mathscr{F}$.
My confusion here is that I don't know what it means to take the pullback sheaf along a sheaf morphism. Is this just asking to take the preimage on any open subset of the sections $i_*i^* \mathscr{F}(U)$ to obtain a subsheaf of $\mathscr{G}$? What does $\rho^{-1}$ mean?
EDIT: Below there is a mistake. It essentially spoils the whole solution but I will leave it. $\rho$ does not need to be an isomorphism, I misquoted the exercise.
This is how I thought to solve this, although it seems too easy to be correct so I think I made a mistake. In the previous part, you show that $i_*\mathscr{F}$ is quasi-coherent and find a maximal coherent subsheaf $\mathscr{F}' \subseteq i_* \mathscr{F}$. I then showed that this sheaf turns out to restrict to $\mathscr{F}$. As such, the only thing missing in this part is to show that $\mathscr{F}'$ is a subsheaf of $\mathscr{G}$, and to do so, we just need to show that $i_*\mathscr{F}$ is a subsheaf of $\mathscr{G}$.
This follows quickly from the fact that $\rho$ is an isomorphism (Liu exercise V.1.1(b)) since $i$ is an open immersion, and that $i_*$ is left exact, right?
After all, $\mathscr{F} \subseteq i^*\mathscr{G}$ so that applying $i_*$ gives $i_*\mathscr{F} \subseteq i_*i^*\mathscr{G}$ by left exactness. Then composing through the isomorphism $\rho$ we get an injective $i_* \mathscr{F} \to \mathscr{G}$ so that $i_*\mathscr{F}$ is indeed a subsheaf of $\mathscr{G}$.
Is this composition with $\rho$ the intention behind the hint?
Thanks!
I do not know if OP has solved it on his own, but the following is the definition.
Then one can check that it is indeed a sheaf from the commutative diagram: $\require{AMScd}$ \begin{CD} \rho_U^{-1}(\mathcal{H}(U)) @>>> \rho_V^{-1}(\mathcal{H}(V))\\ @VV{\rho_U}V @VV{\rho_V}V\\ \mathcal{H}(U) @>>> \mathcal{H}(V) \end{CD} When $\rho$ is a $\mathcal{O}_X$-module morphism and $\mathcal{F},\mathcal{G},\mathcal{H}$ are $\mathcal{O}_X$-modules, the sheaf $\rho^{-1}\mathcal{H}$ is a $\mathcal{O}_X$-submodule of $\mathcal{F}$.