I have the following term $ [M \times [H \times M]] $ under the time derivative. After using the rule of the derivate of the product I get: $$[\dot{M} \times [H \times M]]+[M \times [\dot{H} \times M]]+[M \times [H \times \dot{M}]]$$ let skip the middle term. Now we have 2 terms, $$ X = [\dot{M} \times [H \times M]]+[M \times [H \times \dot{M}]]$$ applying bac-cab rule I get: $$H(\dot{M},M) - M(\dot{M},H) + H(M,\dot{M}) - \dot{M}(M,H)$$ Now I have some problems. As far as I understand there is no need to be precize in which terms should be calculated first (for instance, scalar product of 2 last terms or scalar product of 2 first terms). May I do like this? $$H(\dot{M},M) - M(\dot{M},H) + H(M,\dot{M}) - (M,H)\dot{M}$$ I should get something looking like $2\dot{M}(H,M)$. Nevertheless I do see only that everything collapse.
Found one more thing. Such a result I need, should be get as $[H \times [\dot{M} \times M]] - X$ .
Indeed the equality you have written is correct. Let us use the identity
$a \times (b \times c) = b (a,c) - c(a,b)$
we get that $ H \times (\dot{M} \times M) = \dot{M} (H,M) - M (H, \dot{M})$
Substituting your answer we get:
$$\dot{M} (H,M) - M (H, \dot{M}) - H(\dot{M},M)+ M (\dot{M},H)-H(M,\dot{M})+\dot{M}(M,H)$$
On account of the symmetry of the inner product $(X,Y) = (Y,X)$ the only terms that survive cancellation are
$$ 2\dot{M} (H,M) - 2 H (\dot{M},M) $$ which is your expected answer.