Let $C$ be complex projective curve (Cohen-Macaulay at least). Let $\mathcal{F}$ and $\mathcal{G}$ be coherent sheaves on $C$.
Is there any way to express the degree of $\underline{Hom}_{\mathcal{O}_C}(\mathcal{F},\mathcal{G})$ in terms of the degrees of $\mathcal{F}$ and $\mathcal{G}$?
For locally free sheaves this is obvious but I couldn't prove a generalization to the coherent case.
No, the degree may depend not only on the degrees (and ranks) of $F$ and $G$. For instance, let $C$ be a smooth curve, $F = O_x$ and $G = O_y$. Then $$ \mathcal{H}om(F,G) = \begin{cases} O_x, & \text{if $x = y$},\\ 0, & \text{otherwise} \end{cases} $$ and its degree is 1 in the first case and 0 in the other, while the degree of $F$ and $G$ is equal to 1 (and rank equals 0) in both cases.