I am running into a situation where I have two 4-vectors:
$$ V=(V_0,V_1,V_2,V_3)\\ X=(X_0,X_1,X_2,X_3) $$
and the following relation:
$$ -V_0^2 + V_1^2 + V_2^2 + V_3^2 + X_0^2 - X_1^2 - X_2^2 - X_3^2 - 2 \sqrt{(V_0 X_1 - V_1 X_0)^2 + (V_0 X_2 - V_2 X_0)^2 + (V_0 X_3 - V_3 X_0)^2 - (V_1 X_2 - V_2 X_1)^2 - (V_1 X_3 - V_3 X_1)^2 - (V_3 X_2 - V_2 X_3)^2} $$
The first part can be re-written using the dot product:
$$ V\cdot V - X \cdot X - 2 \sqrt{(V_0 X_1 - V_1 X_0)^2 + (V_0 X_2 - V_2 X_0)^2 + (V_0 X_3 - V_3 X_0)^2 - (V_1 X_2 - V_2 X_1)^2 - (V_1 X_3 - V_3 X_1)^2 - (V_3 X_2 - V_2 X_3)^2} $$
but I cannot identify the pattern of the second part under the square root; is it the det, some combination of the cross/dot products, etc?
The part under the square root is $(V\wedge X)^2=\left(\frac{VX-XV}{2}\right)^2$, where the multiplication is the geometric product in the spacetime algebra (with signature $-+++$).
Given the two vectors
$$V=V_0e_0+V_1e_1+V_2e_2+V_3e_3$$
$$X=X_0e_0+X_1e_1+X_2e_2+X_3e_3,$$
their wedge product is a bivector
$$B=B_{01}e_0e_1+B_{02}e_0e_2+B_{03}e_0e_3+B_{12}e_1e_2+B_{13}e_1e_3+B_{23}e_2e_3$$
$$=V\wedge X=(V_0X_1-X_0V_1)e_0e_1+(V_0X_2-X_0V_2)e_0e_2+(V_0X_3-X_0V_3)e_0e_3\\+(V_1X_2-X_1V_2)e_1e_2+(V_1X_3-X_1V_3)e_1e_3+(V_2X_3-X_2V_3)e_2e_3.$$
The square of a bivector generally has several parts of different grades; but this is a simple bivector, so $B\wedge B=0$, and
$$B^2=B\bullet B=(B_{01})^2+(B_{02})^2+(B_{03})^2-(B_{12})^2-(B_{13})^2-(B_{23})^2.$$
A dot product of wedge products of vectors (I'll use $V_1,V_2,\cdots,V_r$ as several vectors, not components of one vector) can also be written as a determinant:
$$(V_1\wedge V_2\wedge\cdots\wedge V_r)\bullet(W_r\wedge\cdots\wedge W_2\wedge W_1)=\det\begin{bmatrix}V_1\cdot W_1&V_1\cdot W_2&\cdots&V_1\cdot W_r\\ V_2\cdot W_1&V_2\cdot W_2&\cdots&V_2\cdot W_r\\ \vdots&\vdots&\ddots&\vdots\\ V_r\cdot W_1&V_r\cdot W_2&\cdots&V_r\cdot W_r\end{bmatrix},$$
so, in particular,
$$(V\wedge X)^2=-(V\wedge X)\bullet(X\wedge V)=-\det\begin{bmatrix}V\cdot V&V\cdot X\\ X\cdot V&X\cdot X\end{bmatrix}$$
$$=-(V\cdot V)(X\cdot X)+(V\cdot X)^2.$$