Suppose we have a unit vector $\hat{\mathbf{x}}$, a position vector $\mathbf r$, and a scalar $\rho$. Given that $|\mathbf{r}/\rho|\ll1$, we can expand the following expression
$$(1+2\hat{\mathbf{x}}\cdot\frac{\mathbf{r}}{\rho}+\frac{\mathbf{r}^2}{\rho^2})^{-3/2}$$
up to second-order, that is,
$$1-3\hat{\mathbf{x}}\cdot\frac{\mathbf{r}}{\rho}-\frac{3}{2}\frac{\mathbf{r}^2}{\rho^2}+\frac{15}{2}(\hat{\mathbf{x}}\cdot\frac{\mathbf{r}}{\rho})+\dots.$$
Do we use the Taylor theorem here or the binomial theorem?
For the taylor, I am not sure how to do the derivatives here, especially with the dot product.
Any help would be appreciated.
You can use the 'natural' Taylor theorem, as you expand in powers of $||\bf{r}/\rho||$. Simply note that all the quantities involved here are scalars, and that by Cauchy-Schwarz inequality, $|\hat{\bf{x}}.\frac{\bf{r}}{\rho}| \leq ||\bf{\hat{x}}|| \times ||\frac{\bf{r}}{\rho}||$.
Then you simply use the Taylor theorem that states that for $x$ close to $0$ and $a \in \mathbb{R}$, $$(1+x)^a = 1 + a x + \frac{a(a-1)}{2}x^2 + \mathcal{O}(x^3)$$.
So I believe the last term in your expansion should actually be $+\frac{15}{8} \left(\hat{\bf{x}}.\frac{\bf{r}}{\rho}\right)^2$ if I am correct.