R.T.P $|| \vec{x} \times \vec{y}||^{2} = ||\vec{x}||^{2}||\vec{y}||^{2} - (\vec{x} \cdot \vec{y})^{2}$ using index notation.
where $\epsilon_{ijk} = \begin{cases} 1 : (i,j,k) \in \lbrace (1,2,3);(2,3,1);(3,1,2) \rbrace \\ -1: (i,j,k) \in \lbrace (1,3,2);(2,1,3);(3,2,1) \rbrace \\ 0: i=j;i=k;j=k \end{cases}$
initially i used the fact that:
$\vec{x} \times \vec{y} = \epsilon_{ijk} \vec{e_{i}} x_{j} y_{k}$
and that
$|| \vec{x} \times \vec{y} || = \epsilon_{ijk} x_{j} y_{k}$
$|| \vec{x} \times \vec{y} ||^{2} = (\epsilon_{ijk} x_{j} y_{k})^{2}$
im unable to split it into 2 terms that is the main issue.
First, observe that the dot product $\vec{x}\cdot\vec{y}$ of two vectors can be written as $$\vec{x}\cdot\vec{y}=\delta_{ij}x_iy_j=x_iy_i,$$ and that the square of the norm $|\vec{x}\rvert^2 = \vec{x}\cdot \vec{x}$. Now, \begin{align}\lvert\vec{x}\times \vec{y}\rvert^2&=(\vec{x}\times \vec{y})\cdot (\vec{x}\times\vec{y})\\ &= (\varepsilon_{ijk}x_jy_k)\:(\varepsilon_{ilm}x_ly_m) \\ &= (\delta_{jl}\delta_{km}-\delta_{jm}\delta_{kl})\:x_jx_ly_ky_m \\ &= x_jx_jy_ky_k-x_jy_jx_ky_k \\ &=|\vec{x}|^2|\vec{y}|^2-(\vec{x}\cdot\vec{y})^2. \end{align} We have also used the following identity: $$\varepsilon_{ijk}\varepsilon_{ilm}=\delta_{jl}\delta_{km}-\delta_{jm}\delta_{kl}.$$