I have a function $f(x)=\sqrt{-ax}$, where $x\in(-K,0)$, and $a\in\mathbb{R}^{++}$. I proceeded to derive $f'(x)$ in two different ways, in each treating $\sqrt{-a}$ and $\sqrt{a}$ as a constant respectively.
First Way: \begin{align} f'(x)=\frac{\sqrt{-a}}{2\sqrt{x}}=\frac{1}{2}\sqrt{\frac{a}{-x}}\ge 0. \end{align}
Second Way: \begin{align} f'(x)=\sqrt{a}\frac{1}{2\sqrt{-x}}(-1)=\frac{-1}{2}\sqrt{\frac{a}{-x}}\le 0. \end{align}
Which is the correct way?
Second way is the correct one. But I would do it like this: $$g(x)=-ax\\g'(x)=-a\\f(y)=\sqrt{y}\\f'(y)=\frac{1}{2\sqrt{y}}\\f'(g(x))=\frac{g'(x)}{q\sqrt{g(x)}}=\frac{-a}{2\sqrt{-ax}}$$ In your first method neither $\sqrt x$ or $\sqrt{-a}$ are real numbers.