This problem might be simple.
How to expand $|(1-H(f))|^2$? I just see one answer in the following:
(A typo: $S_X(f)$ should be outside the $\{\}$).
My answer is $1+|H(f)|^2 + 2|H(f)|$.
However, "$2\text{Re}[H(f)]\neq2|H(f)|$". (One is the lenght of the vector, the other is the length of the projection onto the real line).
How to expand $|(1-H(f))|^2$?
Hint: For complex numbers, $a.a^* = |a|^2$. where $a^*$ is the complex conjugate of $a$.
The error in your step is that you took $$|1-H(f)|^2 = |1-H(f)||1-H(f)| = (1+|H(f)|)(1+|H(f)|)$$
The last equality is wrong and is in fact a $\le$ by triangle inequality.