Let $X$ be a Riemann surface and $\pi: L \rightarrow X$ be a line bundle. If I delete a point $p$ in $X$, then does the line bundle over $X\setminus\{p\}$ become trivial?
Intuitively, I want to say "no". By simply taking local trivialisations $\phi_i: \pi^{-1}(U_i \setminus \{p\}) \rightarrow U_i \setminus \{p\} \times \mathbb{C}$, I would think that you would obtain a nontrivial bundle at the end. But perhaps, I am missing some sort of Riemann Roch argument.
Yes, every holomorphic line bundle on $X$ becomes trivial on $X\setminus\{p\}$.
Much more generally every holomorphic vector bundle on every non-compact Riemann surface is trivial: you can find this remarkable theorem as Theorem 30.4 (page 229) of Forster's Lectures on Riemann Surfaces.
(Independently of whether $X$ is compact or not, $X\setminus\{p\}$ is definitely not compact so that you can apply theorem 30.4 to your $L$ restricted to $X\setminus\{p\}$)