I have the following operator (polar coordinates are used):
$$ \dfrac{\partial}{\partial x} = \cos \theta \dfrac{\partial}{\partial r} - \dfrac{\sin \theta}{r} \dfrac{\partial}{\partial \theta} $$
This operator is squared to give
Explanation of the book: The last two terms come from differentiation of the coefficients.
I know that this can be obtained by differentiating with respect to $x$ and using the product rule, but in what way is it being calculated here?, what rule is being used?
Looking at applying at what the operator twice does to a function $f(r,\theta)$ may be helpful. Applying the operator once yields
$$ \left(\cos\theta\frac{\partial}{\partial r}-\frac{\sin\theta}{r}\frac{\partial}{\partial\theta}\right)f(r,\theta)=\cos\theta \;f_r-\frac{\sin\theta}{r}f_\theta $$ now applying again in the gory detail, $$ \left(\cos\theta\frac{\partial}{\partial r}-\frac{\sin\theta}{r}\frac{\partial}{\partial\theta}\right)\left(\cos\theta \;f_r-\frac{\sin\theta}{r}f_\theta\right)\\ =\cos\theta\frac{\partial}{\partial r}\left(\cos\theta f_r\right)-\cos\theta\frac{\partial}{\partial r}\left(\frac{\sin\theta}{r}f_\theta\right)\\ -\frac{\sin\theta}{r}\frac{\partial}{\partial\theta}(\cos\theta f_r)+ \frac{\sin\theta}{r}\frac{\partial}{\partial\theta}\left(\frac{\sin\theta}{r}f_\theta\right)\\ =\cos^2\theta f_{rr}-\cos\theta\left(-\frac{\sin\theta}{r^2}f_\theta+\frac{\sin\theta}{r}f_{\theta r} \right)\\ -\frac{\sin\theta}{r}\left(-\sin\theta f_r+\cos\theta f_{r\theta} \right)+ \frac{\sin\theta}{r}\left(\frac{\cos\theta}{r}f_\theta+\frac{\sin\theta}{r}f_{\theta\theta}\right) $$ by the product rule. Now, distributing and combining like terms (and assuming the functions we are acting on have equality of mixed partials), $$ \cos^2\theta f_{rr}+2\frac{\cos\theta\sin\theta}{r^2}f_{\theta}- 2\frac{\cos\theta\sin\theta}{r}f_{r\theta}+\frac{\sin^2\theta}{r}f_{r}+ \frac{\sin^2\theta}{r^2}f_{\theta \theta} $$ meaning the square of your operator can be written as $$ \cos^2\theta \frac{\partial^2}{\partial r^2}+2\frac{\cos\theta\sin\theta}{r^2}\frac{\partial}{\partial \theta}- 2\frac{\cos\theta\sin\theta}{r}\frac{\partial^2}{\partial \theta\partial r}+\frac{\sin^2\theta}{r}\frac{\partial}{\partial r}+ \frac{\sin^2\theta}{r^2}\frac{\partial^2}{\partial\theta^2} $$ matching your book.