If the roots $\alpha$ and $\beta$ of the equation, $x^2-\sqrt2x+c=0$ are complex for some real numbers $c\ne 1$ and $|\frac{\alpha-\beta}{1-\alpha\beta}|=1$ then a value of $c$ is
Squaring both sides, I get $$(\alpha+\beta)^2-4\alpha\beta=(1-\alpha\beta)^2$$
Putting $\alpha+\beta=\sqrt2$ and $\alpha\beta=c$, I get $$c^2+2c-1=0\implies c=-1\pm\sqrt2$$.
Though, the answer is given as $3-\sqrt6$.
Also, I wonder if the roots being complex have any bearing on the solution.
On
Notice, you can't take squares of $\left|\frac{\alpha-\beta}{1-\alpha\beta}\right|=1$ in this case as $(\alpha-\beta)$ is pure imaginary
The roots $\alpha$ & $\beta$ of quadratic equation: $x^2-\sqrt2x+c=0$ will be complex only if $B^2-4AC<0$ i.e. $$(-\sqrt2)^2-4(1)(c)<0\iff c>\frac12$$ $$\alpha+\beta=\frac{-(-\sqrt2)}{1}=\sqrt2 \quad \text{&}\quad \alpha\beta=c$$ It's worth noticing that the difference of complex roots $\alpha-\beta$ will be pure imaginary as they are conjugate $$\therefore \alpha-\beta=\pm\sqrt{(\alpha+\beta)^2-4\alpha\beta}=\pm\sqrt{(\sqrt2)^2-4c}=\pm\sqrt{2-4c}=\pm i\sqrt{4c-2}$$ Setting corresponding values in $\left|\frac{\alpha-\beta}{1-\alpha\beta}\right|=1$, $$\left|\frac{\pm i\sqrt{4c-2}}{1-c}\right|=1$$ $$\sqrt{4c-2}=|1-c|$$ Taking squares as both the sides are positive, $$(\sqrt{4c-2})^2=(1-c)^2$$ $$c^2-6c+3=0$$ $$c=3\pm\sqrt6$$ Both the above real values of $c$ duly satisfy, $c\in \left(\frac12,1\right)\cup \left(1,\infty\right)$
With the help of the comments above, I am able to solve the question. Here is my solution:
Since the roots are complex, so when we square the given expression, we need to use $|z|^2=z\overline z$. Thus, $$\frac{\alpha-\beta}{1-\alpha\beta}\cdot\frac{\overline\alpha-\overline\beta}{1-\overline\alpha\overline\beta}=1$$
And since, complex roots are conjugate of each other. So, by replacing $\overline\alpha$ with $\beta$ and $\overline\beta$ with $\alpha$, we get $$(\frac{\alpha-\beta}{1-\alpha\beta})^2=-1$$
Thus, quadratic in $c$ becomes $$c^2-6c+3=0$$
Thus, $$c=3\pm\sqrt6$$