This is section 19.8 in Vakil's notes. Let $C$ be a nonhyperelliptic curve of genus 4, and $\omega_c$ be its dualizing sheaf. Then $h^0(C, \omega_C) = 4$, and defines a closed embedding of $C$ into $\mathbb{P}^3$. By Riemann-Roch, we can calculate $h^0(C, \omega_C^{\otimes 2}) = 9$. We have the restriction map $H^0(\mathbb{P}^3, \mathscr{O}(2)) \to H^0(C, \omega_C^{\otimes 2})$, and we know that dim Sym$^2\Gamma(C, \omega_C) = \binom{4+1}{2} = 10$. Then the book says this means there is at least one quadric surface in $\mathbb{P}^3$ that contains $C$.
I don't understand how the conclusion follows. In particular, I think I don't understand what $H^0(C, \omega_C^{\otimes 2})$ and Sym$^2\Gamma(C, \omega_C)$ mean geometrically.
So first of all, $H^0=\Gamma$, both denotes the global sections functor.
The closed embedding defined by $\omega_C$ is, when we put it really hands-on, constructed as follows: Take $\alpha_0,\alpha_1,\alpha_2,\alpha_3\in H^0(C,\omega_C)$ linearly independent (a basis if you like) and map \begin{align*} \alpha:C&\longrightarrow\Bbb P^3 \\ x &\longmapsto [\alpha_0(x):\alpha_1(x):\alpha_2(x):\alpha_3(x)] \end{align*} Another way to express this is that a homogeneous degree $d$ polynomial on $\Bbb P^3$ gets pulled back to a homogeneous polynomial in the $\alpha_i$.
Let me set $\mathscr O := \mathscr O_{\Bbb P_3}$. The embedding $\alpha$ gives you a comorphism $\alpha^\sharp\colon \mathscr O\to\mathscr O_C$, essentially turning $\mathscr O_C$ into an $\mathscr O$-module, and consequently turning $\omega_C$ into an $\mathscr O$-module as well. Note that you should think of $\mathscr O_C$ as a quotient of $\mathscr O$, because this map corresponds to a closed embedding. One can show that, when $\alpha$ is a closed embeedding, it yields an isomorphism $\alpha^\ast\mathscr O(1)\cong\omega_C$ of $\mathscr O$-modules. Essentially, it turns $\omega_C$ into $\mathscr O_C(1)$. In particular, $$ \mathscr O_C(2) = \mathscr O_C(1)\otimes_{\mathscr O_C}\mathscr O_C(1) = \omega_C \otimes_{\mathscr O_C} \omega_C = \omega_C^{\otimes2} $$ is the bundle of degree $2$ forms on the embedded curve $C$.
Let us make the language a bit more concrete here. Say $S=\Bbb C[x_0,x_1,x_2,x_3]$ is the homogeneous coordinate ring of projective space and $R$ is the projective coordinate ring of the embedded curve $C\subseteq\Bbb P^3$. Note that $R$ is a graded $\Bbb C$-algebra which is generated in degree $1$. Then, the above is simply a fancy way of saying that $R$ is an $S$-module, so $\mathscr O_C=R^\sim$ is an $\mathscr O=S^\sim$-module, and we have $R_1\cong S_1$ as $\mathscr O(\Bbb P^3)=\Bbb C$-modules, i.e. as vector spaces. Here, $\omega_C=(R[1])^\sim$, so $H^0(C,\omega_C)=R_1$. In particular, $$ S=\operatorname{Sym}S_1=\operatorname{Sym}R_1=\operatorname{Sym} H^0(C,\omega_C), $$ so you can reconstruct the space in which you embedded the curve from the line bundle $\omega_C$ alone, which is likely the reason that $S_2$ is referred to as $\operatorname{Sym}^2 H^0(C,\omega_C)$ in the notes.
You know that $\dim_{\Bbb C}(S_2) = 10$ and $\dim_{\Bbb C}(R_2)=h^0(C,\mathscr O_C(2))=h^0(C,\omega_C^{\otimes2})=9$, so you know that in the kernel of the map $S\to R$, there must be a homogeneous polynomial $f$ of degree $2$. This is an element of the vanishing ideal of $C$, so $C$ is contained in $Z(f)$.