Let $\hat{x}$ be the unit vector along the positive $x$-axis. Let $\bar{q}$ be the position vector in polar coordinates, and $\dot{\bar{q}}$ the velocity vector. Velocity in polar coordinates is given as $$\dot{r}e_r+r\dot{\theta}e_{\theta}$$
where $e_r,e_{\theta}$ are the radial and angular unit vectors. I'm looking at the triple cross product $$\vert \hat{x}\times(\bar{q}\times\dot{\bar{q}})\vert^2=\vert(\hat{x}\cdot\dot{\bar{q}})\bar{q}-(\hat{x}\cdot\bar{q})\dot{\bar{q}}\vert^2$$
and for some reason this should be equal to $$\vert\dot{r}\bar{q}\cos\theta-r\dot{\theta}\bar{q}\sin\theta-r\dot{\bar{q}}\cos\theta\vert^2$$
How do we arrive to this result? I think the logic here should be the following. We have to write $\hat{x}$ in polar coordinates, as in $\hat{x} = (\cos\theta,\sin\theta)$ since it has magnitude one. Then if $\dot{\bar{q}}=(\dot{r},r\dot{\theta})$, we have that $\hat{x} \cdot \dot{\bar{q}}=\dot{r}\bar{q}\cos\theta$.
But the order of the components matters since we're doing dot products, and the components in $\dot{\bar{q}}=(\dot{r},r\dot{\theta})$ are not necessarily written in this order. I've never really thought about this. Maybe its just a matter of convention. But ultimately this reasoning does not explain where the two negative terms come from. I haven't really explored the cross and dot products, so apologies if I'm missing some obvious identities here.
Instead of writing things in terms of pairs in $\Bbb R^2$, write them in terms of vectors: $$\hat e_r = \cos\theta \hat x + \sin\theta\hat y\\\hat e_\theta = -\sin\theta \hat x + \cos\theta \hat y$$
By the BAC-CAB rule, $$\hat x \times (\bar q \times \dot{\bar q}) = (\hat x \cdot \bar q)\dot{\bar q}-(\hat x \cdot \dot{\bar q})\bar q$$ $\bar q = r\hat e_r$, so $$\hat x \cdot \bar q = r(\hat x \cdot \hat e_r) = r\cos \theta$$ And $$\\\hat x \cdot \dot{\bar q} = \dot r \hat x \cdot \hat e_r + r\dot\theta\hat x \cdot \hat e_\theta = \dot r\cos \theta - r\dot\theta\sin\theta$$ Thus $$\begin{align}\hat x \times (\bar q \times \dot{\bar q}) &= (r\cos \theta)\dot{\bar q} - (\dot r\cos\theta - r\dot\theta\sin\theta)\bar q\\&= r\dot\theta\sin\theta\bar q - \dot r\cos\theta\bar q + r\cos\theta\dot{\bar q}\end{align}$$
This is the exact opposite of the vector inside the norm in your formula. But since norms do not care about signs, the two norms are the same. $$\left|\hat x \times (\bar q \times \dot{\bar q})\right| = \left|\dot r\cos\theta\bar q - r\dot\theta\sin\theta\bar q - r\cos\theta\dot{\bar q}\right|$$
That said, expanding this farther gives $\left|\hat x \times (\bar q \times \dot{\bar q})\right| = \left|r^2\dot\theta\right|$, which can be obtained more simply by computing $$\bar q \times \dot{\bar q} = r^2\dot\theta \hat z$$