My specific question is as follows:
In $H^\ast(BSO(3);\mathbb{Z}/2\mathbb{Z})\cong \mathbb{Z}/2\mathbb{Z}[w_2,w_3]$, what is $Sq^2(w_3)$?
I seem to have proven (below) that the answer is simultaneously $0$ and $w_2 w_3$. Since $w_2 w_3\neq 0 $ in this ring, obviously, at least one proof is flawed.
Proposition 1: We have $Sq^2(w_3) = 0$.
Proof Consider the tautological bundle $\xi$ over $BSO(3)$. Restricting to the sphere bundle, we obtain a bundle $S^2\rightarrow E\rightarrow BSO(3)$. From this question, $E$ has the homotopy type of $BSO(2)$.
Since $BSO(3)$ is simply connected, there are no local coefficients to worry about in the spectral sequence associated to the fibration $S^2\rightarrow BSO(2)\rightarrow BSO(3)$. As $BSO(2)\cong \mathbb{C}P^\infty$, one easily sees that if $x\in H^2(S^2;\mathbb{Z}/2\mathbb{Z})\cong \mathbb{Z}/2\mathbb{Z}$ is non-trivial, that $d_2(x) = 0$ and $d_3(x) = w_3$.
Because $d_2(x) = 0$, $x$ is transgressive. Applying $Sq^2$ both sides of the equation $d_3(x) = w_3$ and using the fact that Steenrod squares commute with differentials in the spectral sequence (see McCleary's Users guide to spectral sequences, Corollary 6.9), we find that $$Sq^2(w_3) = Sq^2(d_3(x)) = d_5(Sq^2(x)) = 0,$$ since $Sq^2(x)\in H^4(S^2;\mathbb{Z}/2\mathbb{Z}) = 0.$ $\square$
Proposition 2: We have $Sq^2(w_3) = w_2 w_3$.
Proof: Consider the inclusion $i:BSO(3)\rightarrow BSO$. The induced map $H^\ast(i):H^\ast(BSO;\mathbb{Z}/2\mathbb{Z})\cong \mathbb{Z}/2\mathbb{Z}[w_2',w_3',w_4',....]\rightarrow H^\ast(BSO(3);\mathbb{Z}/2\mathbb{Z})$ maps $w_2'$ to $w_2$, $w_3'$ to $w_3$, and all other $w_j$ to $0$.
In $BSO$, we have the Wu formula $Sq^2(w_3') = w_2' w_3' + w_5'$ (See Milnor-Stasheff's Characteristic classes, Exercise 8.A). Using naturality of Streenrod squares, we find \begin{align*} Sq^2(w_3) &= Sq^2(H^\ast(i)(w_3'))\\ &= H^\ast(i)(Sq^2(w_3'))\\ &= H^\ast(i)(w_2'w_3' + w_5')\\ &= w_2 w_3.\end{align*} $\square$
Where is my mistake?
Surely Proposition 2 must be correct, as the argument is simplest and hardest to doubt. And indeed it is.
One must be suspicious of a spectral sequence argument that gives you something on the nose. Sometimes it works, but you must be careful that you haven't mistaken "x is zero" for "x is zero in the associated graded", aka "x lies in higher filtration".
In particular, the notation in McCleary's Theorem 6.9 has a small bit of evil to it, as hidden after the statement of Proposition 6.5. When he says 'equal', he means 'as elements of $H^*(B)/\text{ker}(p^*)$'.
In your case, $\text{ker}(p^*) = (w_3)$. So the equality in symbol in your position 1 ought to simply say $Sq^2(w_3) \equiv 0 \pmod{w_3}$. This does not contradict the much better result of Proposition 2.
(To see why Corollary 6.9 is self-contained, note that it's referring to equality of elements on some page of a spectral sequence rather than equality of elements in some particular cohomology group.)