I want to let $X$ be a K3 surface, with $Y \subset X$ a smooth curve with genus $g$. Since $Y$ is a hypersurface, we have a line bundle $\mathcal{O}(Y)$ on $X$. I'm curious how to prove the following statement,
$\int_{X} c_{1}^{2}(\mathcal{O}(Y)) = 2g-2$
I feel like I most likely need to use some exact sequence, perhaps like,
$0 \to \mathcal{O}(-Y) \to \mathcal{O}_{X} \to \mathcal{O}_{Y} \to 0$,
but I'm not totally sure how. Also, perhaps there are other ways to get at this formula, but I'm hoping for a rather direct argument specifically pertaining to this integral. Much thanks in advance.
You are basically correct. What you're looking for is the adjunction formula, which states that (for a smooth hypersufrace $Y \subset X$ $$ K_Y = K_X|_Y \otimes N_{Y/X} $$ where $N_{Y/X}$ is the normal bundle of $Y$ in $X$; in the case of a line bundle, we have that $N_{Y/X} \cong \mathscr{O}_X(Y)|_Y$. Since the canonical bundle of a K3 surface is trivial, we have that $K_Y = \mathscr{O}_X(Y)|_Y$
In particular, since $$ \int_X c_1\mathscr{O}_X(Y)^2 = \deg \mathscr{O}_X(Y)|_Y $$ by Poincaré duality, the claim follows.