Find the possible values of $p$ for which $p$ and $q$ are the solutions of the equation $$x^2+px+q=0, \ \ \{x,p,q\}\in\ \mathbb{R}.$$
Plugging $p$ and $q$ into the given equation $\begin{cases} 2p^2+q=0 \\[2ex] q^2+pq+q=0 \end{cases}$
gives $p=\{0,1,-\dfrac{1}{2}\},$ which is the answer given by my book.
On the other hand, using Vieta's formula for the sum and product of roots $\begin{cases} p+q=-p \\[2ex] pq=q \end{cases}$
gives $p=\{0,1\}.$ I am highly confused as to why Vieta's formula fails.
In short, the difference is because the Vieta formulae assume that $p$ and $q$ are different roots of the equation (in the sense of corresponding to different factors — not necessarily in the sense of being numerically unequal), while your second system only assumes that $p$ and $q$ both solve the equation.
To be more explicit, let's solve your second system to see what's going on. Subtracting the two equations from each other gives $$ q^2+pq-2p^2=0\\ (q-p)(q+2p)=0 $$
So either $q=p$ or $q=-2p$.
If $q=-2p$, we get $2p^2-2p=0$, so either $p=q=0$ or $p=1$, $q=-2$. Note that both of these are solutions to the Vieta system as well.
If $q=p$, we get $2p^2+p=0$, so either $p=q=0$ (again) or $p=q=-1/2$. This last solution does not satisfy the Vieta formulae. Why not? Because it means the original equation was $$ x^2-\frac{1}{2}x-\frac{1}{2}=0 $$ which has solutions $x=-\frac{1}{2}$ and $x=1$. So in this case both $p$ and $q$ are roots of $x^2+px+q$, but they aren't "the" roots of $x^2+px+q$ (since they're both the same root).