In polar coordinates $\rho$ is the radius vector and $\theta$ is the angle the radius vector makes with the positive x axis. Let $\psi$ be the angle that the radius vector makes with the tangent to the curve, $\psi = \arctan(\rho/\rho')$.
In deriving the formula for curvature $K$, $d\psi/d\theta$ is found to be
$$
\frac{(\rho')^2-\rho\rho''}{(\rho')^2 +\rho^2}.
$$
Why is it that $d\psi/d\theta$ does not equal
$$
\frac{\rho'((\rho')^2-\rho\rho'')}{(\rho')^2 +\rho^2},
$$
considering that
$$
\frac{d\psi}{d\theta} = \frac{d\psi}{d\rho} \frac{d\rho}{d\theta}
$$
and $d\rho/d\theta = \rho'$?
First of all, $\rho$ is not the radius vector; it's the polar distance. Everything here is a function of $\theta$; you're trying to introduce a spurious application of the chain rule. $\psi$, in particular, is not a function of $\rho$ — you could view it as a function of the two (not so independent) variables $\rho$ and $\rho'$, if you wish, but we don't want to be doing multivariable calculus and the multivariable chain rule here. Just write $\psi(\theta) = \arctan\left(\frac{\rho(\theta)}{\rho'(\theta)}\right)$, and you get their result immediately.
In particular, you should ask yourself at the outside what the $^\prime$ and $^{\prime\prime}$ symbols mean. The only way the whole discussion makes sense is for $^\prime$ always to mean $d/d\theta$.