so I really can't see what I am doing wrong. I want to use this formula:
$a\times (b\times c) = b(a\cdot c) - c(a\cdot b)$
Calculate the rotation of $v(x,y,z)=(x,y,z)^T \times \omega$ with $\omega \in \mathbb R^3$
Solution:
$a\cdot c=\nabla\cdot \omega=0$
and
$a\cdot b = \nabla \cdot (x,y,z)^T= \partial_x x + \partial_y y + \partial_z z = 3$
so we get $-\omega 3=-3\omega$
The actual solution (which I do get by direct calculation) is: $-2\omega$
On
On Wikipedia you can see that the formula for the curl of a cross product is given by $$ \nabla \times (\mathbf{A} \times \mathbf{B}) = \mathbf{A}\ (\nabla \cdot \mathbf{B}) - \mathbf{B}\ (\nabla \cdot \mathbf{A}) + (\mathbf{B} \cdot \nabla) \mathbf{A} - (\mathbf{A} \cdot \nabla) \mathbf{B} . $$
Applying this on your case gives $$\begin{align} \nabla \times (\mathbf{x} \times \mathbf{\omega}) &= \mathbf{x}\ (\nabla \cdot \mathbf{\omega}) - \mathbf{\omega}\ (\nabla \cdot \mathbf{x}) + (\mathbf{\omega} \cdot \nabla) \mathbf{x} - (\mathbf{x} \cdot \nabla) \mathbf{\omega} \\ &= \mathbf{x} \ 0 - \mathbf{\omega}\ 3 + \mathbf{\omega} - \mathbf{0} = -2 \mathbf{\omega} . \end{align}$$
If you want to generalise formulae with vectors having commuting components, make sure to write the usual formula so as to preserve the order in products. For example, here the $i$th component of the RHS is $\sum_j a_j (b_i c_j-b_j c_i)$, once we impose the $abc$ order of the LHS. Now you know the generalisation off the top of your head. For $a_i=\partial_i$, the result's $i$th component is $$\sum_j \partial_j (b_i c_j-b_j c_i)=b_i\nabla\cdot c+c\cdot\nabla b_i-(\nabla\cdot b) c-b\cdot\nabla c_i.$$The vector, in other words, is $$\vec{b}(\nabla\cdot\vec{c})-(\nabla\cdot\vec{b})\vec{c}+(\vec{c}\cdot\nabla)\vec{b}-(\vec{b}\cdot\nabla)\vec{c}.$$(I've swapped the middle terms to mirror @md2perpe's quoted result, but the letters $b,\,c$ still need to be changed to $A,\,B$).