I need to solve the following equation:
$$ \frac{d m}{d t}=-m\wedge b-\alpha m\wedge (m\wedge b), $$ where $b$ is constant
However, I was instructed specifically not to separate the calculation into a system of equations (which would not be linear and would be pretty complicated). I was told, instead, that there is a smarter way to do it.
I managed to do some of it. I found the angle between m and b. And now I'm stuck.
Here is what I did so far:
$$\frac{dm}{dt} \cdot m = 0 \Rightarrow \frac{d\vert m \vert}{dt} = 0$$
Naming $u=|m\wedge b|^2$ and $v=m\cdot b$, we instantly get the relation: $$u+v^2=k^2$$ Where $k$ is a constant independent of time and of value $k=|b||m|$. We can rewrite the original equation as the following (triple product):
$$ \frac{d m}{d t}=-m\wedge b-\alpha (m\cdot b)m + \alpha (m\cdot m)b $$
Then,
$$\frac{dm}{dt} \cdot b = - \alpha(m\cdot b)(m\cdot b) + \alpha(m\cdot m)(b\cdot b)$$
$$\frac{dv}{dt} = - \alpha v^2 + \alpha k^2 $$
We also retrieve that
$$\frac{dv}{dt} = - \alpha u $$
To solve the differential equation in v, we divide both sides by $ (k^2 - v^2)$, and then integrate.
$$ln|v-k| - ln|v+k| = -2\alpha kt - 2C$$
$$|\frac{v-k}{v+k}| = e^{-2\alpha k t - 2C}$$
We know that $k>0$ and $v>0$, and that always $v<=k$, then $|\frac{v-k}{v+k}| = -\frac{v-k}{v+k}$ and the equation yields:
$$v = -k\frac{1-e^{-2\alpha k t - 2C}}{1+e^{-2\alpha k t - 2C}} = -k\tanh(\alpha k t + C)$$
$$u = \frac{k^2}{\cosh^2(\alpha k t + C)}$$
We can extract:
$$\cos(\phi) = \frac{k}{|m||b|\cosh(\alpha k t + C)}$$
$$\sin(\phi) = \frac{k \tanh(\alpha k t + C)}{|m||b|}$$
But $|m|$ and $|b|$ are constants.
After the realization that $|m|$ is constant, the rest of your computation could be equivalently expressed in spherical coordinates, which I think makes it more transparent. (Perhaps the warning you were given was about Cartesian coordinates?)
Indeed, choose the $z$-axis along $b$. The term $-m\wedge b$ is horizontal, i.e., it pertains to the change of longitude $\theta$. The term $-\alpha m\wedge (m\wedge b)$ is orthogonal to it, so it affects only the change of latitude $\phi$. I get $$ \theta' = - |b|\sin \phi,\quad \phi' = -\alpha |m| |b| \sin \phi $$ Up to possible sign errors. In any case you already integrated the second equation for $\phi$. The function $\theta$ is directly extressed in terms of $\int \sin\phi(t)\,dt$, which may or may not simplify.