A (discrete) LTI system has the frequency response
$$ H(e^{j\omega}) = \frac{1-1.25e^{-jw}}{1-0.8e^{-j\omega}} $$
Show that $|H(e^{jw})|=G^2$, where G is a constant. Determine the constant $G$.
Expanding the exponentials to the form $e^{j\theta} = cos\theta + jsin\theta$, gives the following expression:
$$ \frac{(1 - 1.25\cos{\omega}) + j1.25\sin{\omega}}{(1 - .8\cos{\omega})+ j0.8\sin{\omega}} $$
the square of the magnitude is
$$ \frac{1-2.5\cos{\omega}+2.25}{1 - 1.6\cos{\omega}+0.64} $$
Clearly, putting in different values of $\omega$ will give different values of squared magnitudes in the expression above.
I feel that my approach is not correct; where did I go wrong?
Your calculation of the squared magnitude is wrong because $(1.25)^2\neq 2.25$. The magnitude of the given transfer function is indeed constant. This can be shown in a general way as follows:
$$\begin{align}H(z)&=\frac{1-az^{-1}}{1-\frac{1}{a}z^{-1}}\\&=a\frac{1-az^{-1}}{a-z^{-1}}\\&=-az\frac{P(z^{-1})}{P(z)}\tag{1}\end{align}$$
On the unit circle $z=e^{j\omega}$ we have $|z|=1$ and $|P(z)|=|P(z^{-1})|$, and, consequently, $|H(z)|=|a|$. Hence, for the given example we obtain $G=a=1.25$.