Let $C$ be a nonsingular plane curve of degree 5 over $\mathbb{C}$, as in the question, I want to show that $C$ has no linear system of dimension 1 and degree 3, that is has no $g^{1}_{3}$.
First, note $g(C)=6$. Suppose $C$ has $g_{3}^{1}$, then there exists an effective divsior $D$, such that $\mathrm{deg}(D)=3$ and $\mathrm{dim} |D|\geq1$. By Riemann-Roch, we have $\mathrm{dim}|D|-\mathrm{dim}|K-D|=-2$. Thus $\mathrm{dim}|K-D|=\mathrm{dim}|D|+2\geq 3$. But then how to deduce this is impossible?
Note here actually I did not use that $C$ is a plane curve. This condition should be necassary. Could you help me?
Assume $C$ is trigonal and let $C \subset \mathbb{P}^{g-1}$ be the canonical embedding. Then the intersection in $\mathbb{P}^{g-1}$ of all quadrics passing through $C$ is a rational scroll. On the other hand, if $C$ is a plane quintic, its canonical embedding is equal to the composition $$ C \to \mathbb{P}^2 \to \mathbb{P}^5, $$ where the second arrow is the double Veronese embedding, and it is easy to check that the intersection in $\mathbb{P}^5$ of all quadrics passing through $C$ is the Veronese surface. Since it is not a rational scroll, the curve is not trigonal.