$\underline {Background}$:Suppose $X$ be a scheme and $\mathcal F$ and $\mathcal G$ are two sheaves of $\mathcal O_X$ modules.
Also assume that $\exists ${U}$ $ a cover of $X$ by open sets such that each $\mathcal F(U),\mathcal G(U)$ are finitely free $\mathcal O(U)$ modules.
Let $ \exists \phi:\mathcal F\to\mathcal G$ a sheaf of $\mathcal O_X$ modules.
$\underline {Question}$:what is the meaning of the statement "trace$\phi=0$"
$\underline {Guess}$:Does it mean for all open set $V$ of $X$
$\phi(V):\mathcal F(V)\to \mathcal G(V)$ has trace of the matrix of $\phi(V)=0$
But this makes sense only for $V=U$ because,we only have $\phi(U)$ is a morphism between 2 finitely free $\mathcal O(U)$ modules,and hence its matrix w.r.to canonical bases exists.
So,I would like to know the appropriate definition of trace being $0$ and any text which mentions it clearly.
Any help from anyone is welcome.
As you say, the trace of $\phi$ is only meaningful when $\mathcal{G}=\mathcal{F}$. Slightly more generally, though it still makes sense to talk about the trace of $\phi$ being $0$ if you have a chosen isomorphism $\mathcal{G}\cong\mathcal{F}\otimes\mathcal{L}$ for some line bundle $\mathcal{L}$. Indeed, picking a local trivialization of $\mathcal{L}$, we get local isomorphisms $\mathcal{G}\cong\mathcal{F}$ which we can use to compute the trace. Now of course the trace will depend on the trivialization of $\mathcal{L}$ chosen, but changing the trivialization just multiplies the isomorphism by some invertible section of $\mathcal{O}_X$, and so will not change whether the trace is $0$.
Another way to see it is that a morphism $\mathcal{F}\to\mathcal{F}\otimes\mathcal{L}$ is equivalent to a morphism $\mathcal{L}^\vee\to\mathcal{F}\otimes\mathcal{F}^\vee$. There is a canonical "trace" map $\mathcal{F}\otimes\mathcal{F}^\vee\to\mathcal{O}_X$ (just the evaluation pairing) and so we can compose to get a morphism $\mathcal{L}^\vee\to\mathcal{O}_X$ (or equivalently, a section of $\mathcal{L}$) which we can call the "trace" of $\phi$.