Could someone nudge me in the right direction on how to get the magnitude of
$H(\omega) = (1-\sqrt(2)e^{-j\omega}+e^{-2j\omega}) / (1-.5\sqrt(2)e^{-j\omega}+.25e^{-2j\omega})$
If it was just a two term denominator and numerator I could just use conjugate to get the magnitude but I'm not sure how this quadratic like equation could be solved to get the magnitude. Would I separate the top and bottom then get those magnitudes separately? If so which equations would I use? Could I use $x = b +- sqrt(b^2-4(a)(c))/ 2a?$
If we write this as $$ H(\omega) = {a(\omega) \over b(\omega)}$$ then you can see that $b(\omega) H(\omega) = a(\omega)$. In particular, $\vert a(\omega)\vert = \vert b(\omega)H(\omega) \vert = \vert b(\omega) \vert \cdot \vert H(\omega) \vert$. This is one way to show $$ \vert H(\omega) = {\vert a(\omega) \vert \over \vert b(\omega) \vert } $$ so you can find your answer by taking magnitudes of numerator and denominator separately.
Now since each $a$ and $b$ is a sum the calculation is not completely straightforward; I would multiply by conjugates. That is, since $z \overline{z} = \vert z \vert ^2$, you can try $$ \vert a(\omega) \vert^2 = a(\omega)\overline{a(\omega)} = (1 - \sqrt{2}e^{-j\omega} + e^{-2j\omega})(1 - \sqrt{2}e^{j\omega} + e^{2j\omega}) $$ $$ = 4 - \sqrt{2}(e^{j\omega} + e^{-j\omega}) + (e^{2j\omega}+e^{-2j\omega}) - \sqrt{2}(e^{j\omega}+e^{-j\omega})$$ and use the fact that $\displaystyle \cos(\omega) = {e^{j\omega} + e^{-j\omega} \over 2}$to begin calculating $|a(\omega)|$. Same works for $\vert b(\omega) \vert$.