Take the following example:
$$\vec\nabla\times(\vec A \times \vec B)$$
I assumed that this worked out to:
$$\vec A(\vec\nabla.\vec B) - \vec B(\vec\nabla.\vec A)$$
Where, in both terms, Nabla works in on both $\vec A$ and $\vec B$.
But according to my professor, it works out to the following:
$$\vec A(\vec\nabla.\vec B) - \vec B(\vec A.\vec \nabla)$$
Where, in the both terms, Nabla works in on $\vec B$. (My professor uses arrows to indicate where Nabla works in on, in the first term he also has an arrow pointing to $\vec A$ but it's crossed out and I'm not sure if it's an error or not.)
This may or may not be necessary knowledge, but it's from a physical application of Nabla and cross products. I'm working with the following:
A vector potential $\vec A(r)=\frac{\mu_0}{4\pi}\frac{\vec m \times \vec r}{r^3}$ with $\vec m$ the moment.
The task is to calculate the magnetic induction $\vec B = \vec\nabla \times \vec A$
The first equation from above is
$$\vec\nabla \times(\vec m\times\frac{\vec r}{r^3})$$
Or, in other words, the $\vec A$ I used is $\vec m$ and the $\vec B$ I used is $\frac{\vec r}{r^3}$.
I'm not sure if I'm missing something mathematically or if my problem is related to the physics of the equation, but either way it hasn't really clicked for me how I need to work with Nabla.
Your professor is correct assuming $\vec{A}$ is a constant vector field. To answer your particular problem, since $\vec{m}$ is constant,
\begin{align}\vec{B} &= \frac{\mu_0}{4\pi}\left\{\vec{m}\left(\vec{\nabla} \cdot \frac{\vec{r}}{r^3}\right) - (\vec{m} \cdot \vec{\nabla})\frac{\vec{r}}{r^3}\right\}\\ &= \frac{\mu_0}{4\pi}\left\{\vec{m}\left(\frac{\vec{\nabla}\cdot \vec{r}}{r^3} + \vec{\nabla}\left(\frac{1}{r^3}\right)\cdot \vec{r}\right) - (\vec{m}\cdot \vec{\nabla})\frac{\vec{r}}{r^3}\right\}\\ &= \frac{\mu_0}{4\pi}\left\{\vec{m}\left(\frac{3}{r^3} + \frac{1}{r} \frac{d}{dr}\left(\frac{1}{r^3}\right)\vec{r}\cdot \vec{r}\right) - (\vec{m}\cdot \vec{\nabla})\frac{\vec{r}}{r^3}\right\}\\ &= \frac{\mu_0}{4\pi}\left\{\vec{m}\left(\frac{3}{r^3} - \frac{3}{r^3}\right) - (\vec{m}\cdot \vec{\nabla})\frac{\vec{r}}{r^3}\right\}\\ &=-\frac{\mu_0}{4\pi}(\vec{m}\cdot \vec{\nabla})\frac{\vec{r}}{r^3}. \end{align}
Edit: If $\vec{m} = (m_1,m_2,m_3)$, then
\begin{align}\vec{m}\cdot\vec{\nabla}\frac{x}{r^3} &= \vec{m} \cdot \left(\frac{1}{r^3}\hat{x} - 3x\frac{\vec{r}}{r^5}\right) = \frac{m_1}{r^3} - 3x\frac{\vec{m}\cdot \vec{r}}{r^5},\\ \vec{m}\cdot\vec{\nabla}\frac{y}{r^3} &= \vec{m}\cdot \left(\frac{1}{r^3}\hat{y} - 3y\frac{\vec{r}}{r^5}\right) = \frac{m_2}{r^3} - 3y\frac{\vec{m}\cdot \vec{r}}{r^5},\\ \vec{m}\cdot\vec{\nabla}\frac{z}{r^3} &= \vec{m}\cdot \left(\frac{1}{r^3}\hat{z} - 3z\frac{\vec{r}}{r^5}\right) = \frac{m_3}{r^3} - 3z\frac{\vec{m}\cdot \vec{r}}{r^5},\end{align}
implying that
$$(\vec{m}\cdot \vec{\nabla})\frac{\vec{r}}{r^3} = \frac{\vec{m}}{r^3} - 3\vec{r}\frac{\vec{m}\cdot \vec{r}}{r^5}.$$
Hence
$$\vec{B} = -\frac{\mu_0}{4\pi}(\vec{m}\cdot\vec{\nabla})\frac{\vec{r}}{r^3} = \frac{\mu_0}{4\pi}\left(3\vec{r}\frac{\vec{m}\cdot \vec{r}}{r^5} - \frac{\vec{m}}{r^3}\right).$$