I was trying to solve the problems of Liu's book and wanted to show that if both $F$ ,$G$ are coherent then $Hom(F,G)$ is also coherent... but I realised that I really need to understand the meaning and definition of the sheaf of homomorphisms.... so I was tryinng to understand these:
Let $X$ be a scheme, $F$, $G$ two quasi-coherent $O_X$-modules and $U=\operatorname{Spec} A$ be an affine open subset of $X$. Suppose that $F|_U =\widetilde{M}$ and $G|_U=\widetilde{N}$. I really need some help to show if $\mathcal{Hom}_U(\widetilde{M},\widetilde{N})$ is isomorphic to $\operatorname{Hom}_A(M,N)^\sim$
Is $\mathcal{Hom}(\widetilde{M},\widetilde{N})$ also a quasi coherent sheaf?
Note that $Hom_U(\tilde{M},\tilde{N})$ identifies naturally with $Hom_A(M,N)$. Indeed, given $f : \tilde{M} \rightarrow \tilde{N}$, its action on the global sections is a $A$-linear map $M \rightarrow N$. Conversely, given $g: M \rightarrow N$, we can consider the maps $g_{D(a)}: M_a=\tilde{M}(D(a)) \rightarrow N_a=\tilde{N}(D(a))$ which are the localizations of $g$ at $a$; they do glue together to form a sheaf morphism $\tilde{g}: \tilde{M} \rightarrow \tilde{N}$.
Now, let $H$ be the sheaf which is the internal Hom of $\tilde{M}$ and $\tilde{N}$ over $U$. By definition, if $V \subset U$ is principal (ie is $D(a)$), $H(V)$ is the set of homomorphisms $\tilde{M}_{|V} \rightarrow \tilde{N}_{|V}$, so by the above identifies with the set of homomorphisms $\tilde{M}(V) \rightarrow \tilde{N}(V)$, ie of homomorphisms $M_a \rightarrow N_a$. However, $\tilde{Hom_A(M,N)}(V)$ is $Hom(M,N) \otimes \mathcal{O}(V)=Hom(M,N)_a$.
The question thus becomes: how to compare $Hom(M,N)_a$ and $Hom(M_a,N_a)$?
It’s easy to see that we have a $A$-linear map $Hom(M,N)_a \rightarrow Hom(M_a,N_a)$.
You can check that this map is injective if $M$ is finitely generated – and that it isn’t in general if $M$ isn’t (hint: consider the case $M,N$ free). It is bijective if $M$ is free.
If we have an exact sequence $A^m \rightarrow A^n \rightarrow M \rightarrow 0$ (ie $M$ finitely presented) then we have a natural morphism between the exact sequences $$0 \rightarrow Hom(M,N)_a \rightarrow Hom(A^n,N)_a \rightarrow Hom(A^m,N)_a$$ and $$0 \rightarrow Hom(M_a,N_a) \rightarrow Hom((A_a)^n,N_a) \rightarrow Hom((A_a)^m,N_a).$$
The two rightmost vertical arrows are isomorphisms, so $Hom(M,N)_a \rightarrow Hom(M_a, N_a)$ is an isomorphism.
As a conclusion, the inner Hom of $\tilde{M},\tilde{N}$ is equal to $\tilde{Hom_A(M,N)}$ when $M$ is finitely presented. When $M$ is only finitely generated, the latter is a subsheaf of the former.