I'm trying to solve the equation $$ E(x)=\frac{(1+ix)^{1/2}-(1-ix)^{1/2}}{(1+ix)^{1/2}+(1-ix)^{1/2}} \mod p, $$
where $p$ is a prime and $1+ix$ is a Gaussian integer over $p$. For some values of $x$, there is no solution for the $\sqrt{1+ix}$ in the complex field, i.e., is a quadratic nonresidue. I'm just an engineer student, my knowledge in the area is not great. There is a possibility that the solution is in an extension field like the Hurwitz quaternion?
What you propose is possible to do. Suppose $\,p\,$ is an odd prime. Let $$ E(x):=\frac{(1+ix)^{1/2}-(1-ix)^{1/2}}{(1+ix)^{1/2}+(1-ix)^{1/2}} \pmod p. $$ First, simplify the quotient to get (if $\,x\ne 0$ otherwise $E(0)=0$) $$ E(x) = i \frac{\sqrt{1+x^2}-1}x \pmod p. $$ Here we understand $\,i\,$ to mean a square root of $\,-1\pmod p\,$ if $\,p-1\,$ is a quadratic residue, or else it exists in an extension field. Again, the same problem arises for $\,\sqrt{1+x^2}\,$ if $\,1+x^2\,$ is not a quadratic residue and so use an extension field. Thus, we may need to use two extension fields to make sense of $\,E(x).\,$ Note that $\,E(-x)=-E(x)\,$ for all $\,x.\,$
For example, let $\,p=3.\,$ Define $\,i=\sqrt{-1}=\sqrt{2}\,$ in an extension field of $\,\mathbb F_3.\,$ We already know $\,E(0)=0.\,$ Now $\,E(1)=2+2i\,$ and $\,E(2)=1+i.\,$
Another example, let $\,p=5.\,$ Here $\,-1=2^2=3^2\pmod 5\,$ thus $\,i=2.\,$ Now we need $\,r:=\sqrt{2}\,$ in an extension field of $\,\mathbb F_5\,$ and thus, $$E(0)=0,\; E(1)=3+2r,\; E(2)=4,\; E(3)=1,\; E(4)=2+3r. $$