I was asked to calculate the curl of the vector field $\theta^2\bf e_\theta$ in spherical coordinates. If we use the typical formula for curl in curvilinear coordinates: $$ \nabla\times \bf F = \left|\begin{matrix}h_1\hat{\bf e_1} & h_2\hat{\bf e_2} & h_3\hat{\bf e_3} \\ \displaystyle\frac{\partial}{\partial q_1} & \displaystyle\frac{\partial}{\partial q_2} & \displaystyle\frac{\partial}{\partial q_3} \\ h_1F_1 & h_2F_2 & h_3F_3\end{matrix}\right| $$ where, in spherical coordinates, $(q_1,q_2,q_3) = (r,\theta,\phi)$, and $h_1 = 1$, $h_2 = r$, $h_3 = r\sin\theta$, then we get $$ \nabla\times(\theta^2{\bf e}_\theta) = r\theta^2\sin\theta\bf e_\phi $$
I wanted to check this, so I thought I'd try converting the original vector field into Cartesian coordinates, taking the curl in Cartesian coordinates, and converting back. Using $$ \theta = \arctan\left({\frac{\sqrt{x^2+y^2}}{z}}\right) \\ {\bf e_\theta} = \frac{(x\hat{\bf i} + y\hat{\bf j})z - (x^2+y^2)\hat{\bf k}}{\sqrt{x^2+y^2+z^2}\sqrt{x^2+y^2}} $$ we get $$ \theta^2{\bf e}_\theta = \frac{\arctan^2\left(\frac{\sqrt{x^2+y^2}}{z}\right)}{\sqrt{x^2+y^2}\sqrt{x^2+y^2+z^2}}\left(xz,yz,-(x^2+y^2)\right) $$
After some tedious calculation, this (somewhat remarkably) comes out to $$ \nabla\times{\bf F} = \frac{\arctan^2\left(\frac{\sqrt{x^2+y^2}}{z}\right)}{\sqrt{x^2+y^2}\sqrt{x^2+y^2+z^2}}(-y{\bf\hat{i}}+x{\bf\hat{j}}) $$ This takes some work to convert back to polar coordinates, but without converting the unit vectors back immediately we have $$ \nabla\times{\bf F} = \frac{\theta^2}{r}(-\sin\phi{\bf\hat{i}} + \cos\phi{\bf\hat{j}}) $$ After substituting $$ {\bf\hat{i}} = \sin\theta\cos\phi{\bf\hat{r}}+\cos\theta\cos\phi\hat{\theta}-\sin\phi\hat{\phi} \\ {\bf\hat{j}} = \sin\theta\sin\phi{\bf\hat{r}} + \cos\theta\sin\phi\hat{\theta} + \cos\phi\hat{\phi} $$ into the above formula, we get $$ \frac{\theta^2}r{\bf e}_\theta $$ which is... not the same.
So clearly, my intuition is wrong, or my assumption that these two "curls" in different coordinate systems are the same (since, assuming I didn't make any mistakes in my derivation, this would imply that "spherical curl" is somehow fundamentally different from "Cartesian curl"), or I made a mistake somewhere. Or something else.
Which is it? Can someone guide me in the right direction please?
I see that you miss a factor in your first formula for curl. It should read $$\nabla\times \mathbf F = \frac{1}{h_1 h_2 h_3} \left|\begin{matrix}h_1\hat{\bf e_1} & h_2\hat{\bf e_2} & h_3\hat{\bf e_3} \\ \displaystyle\frac{\partial}{\partial q_1} & \displaystyle\frac{\partial}{\partial q_2} & \displaystyle\frac{\partial}{\partial q_3} \\ h_1F_1 & h_2F_2 & h_3F_3\end{matrix}\right|$$
Thus you get $$\nabla\times\mathbf F = \frac{1}{r^2 \sin\theta} \, r\theta^2\sin\theta \, \mathbf{e}_\phi = \frac{\theta^2}{r} \mathbf{e}_\phi$$
But now I noticed that there is still a difference. In your second attempt, via Cartesian coordinates, you get $\mathbf{e}_\theta$ instead of $\mathbf{e}_\phi$. I believe $\mathbf{e}_\phi$ to be the correct since that's orthogonal to the field.
And now I see that you make some error when you convert $-\sin\phi{\bf\hat{i}} + \cos\phi{\bf\hat{j}}$ back. It should be $\mathbf{e}_\phi$, not $\mathbf{e}_\theta$.