Which one is the correct way of solving this problem?
$i\times\sqrt{-x} = i \times i \times \sqrt{x} = -\sqrt{x}$
or
$i\times\sqrt{-x} = \sqrt{-(-x)} = \sqrt{x}$
Which one is the correct way of solving this problem?
$i\times\sqrt{-x} = i \times i \times \sqrt{x} = -\sqrt{x}$
or
$i\times\sqrt{-x} = \sqrt{-(-x)} = \sqrt{x}$
On
I believe I have found an answer.
The original equation was \begin{equation*} \gamma^2 + \omega^2\mu\epsilon(1-j\frac{\sigma}{\omega\epsilon}) = 0\end{equation*} This is the dispersion relation, and depending on how you define your wave, $\gamma$'s real part can only be either positive or negative. As the accepted answer pointed out there are, however, two solutions to this equation.
On
There is a branch cut discontinuity in the complex plane for the square root function, which is why "equations" like $$1=\sqrt{1\cdot 1}=\sqrt{(-1)(-1)}=\sqrt{-1}\,\sqrt{-1}=i^2=-1$$ fail. When you do $\sqrt{(-1)(-1)}=\sqrt{-1}\,\sqrt{-1}$, you're approaching that discontinuity from two different directions, thus producing an ambiguity. It's very much as if you are jumping off a cliff here, and the equation doesn't know if you're at the top or the bottom of it.
The comments all point to the correct answer. I'll provide a little more background.
First, your question has nothing to do with order of operations. It stems from a misunderstanding of what the $\sqrt{} \ $ symbol does.
When $x$ is a nonnegative real number, $\sqrt{x}$ is the unique nonnegative number whose square is $x$. So $\sqrt{4} = 2$, NOT $\pm 2$. When $x$ is a negative real number there is no real square root.
When you start thinking about complex numbers, every one (other than $0$) turns out to have two square roots. But there is no way to specify one of them as "the" square root and the other as its negative. Both $i$ and $-i$ are square roots of $-1$ but neither one of them is "$\sqrt{-1}$".
In your particular case you want the complex numbers whose square is $i\sigma/\omega\mu$. Assuming those constants are positive, the two square roots are $$ \pm\sqrt{\frac{\sigma}{\omega\mu}} \left( \frac{1 + i}{\sqrt{2}}\right). $$
The moral of the story is that you should avoid the square root symbol when doing complex arithmetic. (So should instructors and authors.)