Let $X,Y$ be two projective variety over an algebraically closed field, then we know $Hom(X,Y)$ has a scheme structure by considering the embedding as a open subscheme of $Hilb(X\times Y)$ using the graph.
We know for each fixed polynomial $P$, the Hilbert scheme with Hilbert polynomial $P$ is projective hence finite type. And it seems that the graph of any $f:X\longrightarrow Y$ is isomorphic to $X$ hence has the same Hilbert polynomial viewed as a closed subscheme of $X \times Y$. However, Hom scheme can have infinitely many component (for example consider $End(E)$ where $E$ is an elliptic curve). So where does my intuition goes wrong? If $X,Y$ are both smooth curves, can we decide all of the connected components?
To define a Hilbert polynomial you need to choose an ample line bundle. A reasonable choice on $X \times Y$ is $L_X \boxtimes L_Y$, where $L_X$ and $L_Y$ are line bundles on $X$ and $Y$. Then if $f \colon X \to Y$ is a morphism, the restriction of $L_X \boxtimes L_Y$ to the graph $\Gamma(f)$ of $f$ corresponds to the line bundle $L_X \otimes f^*L_Y$ on $X$, hence the Hilbert polynomial of $\Gamma(f)$ equals the Hilbert polynomial of $X$ with respect to $L_X \otimes f^*L_Y$. In particular, it does depend on $f$.