The phase retrieval problem consists of recovering phase information from given intensity measurements, as shown in the image below from Deep phase retrieval: analyzing over-parameterization in phase retrieval.
I don't understand why it is a non-convex optimization problem. Can someone please help me on how to prove it? Or is there any paper or book about this topic?

Consider the simple example where $n = 1$, $m = 1$, $a_1 = 1$, $y_1 = 1$, i.e. we are estimating a $1$-dimensional complex variable $z \in \mathbb{C}$ from one measurement $1 = |z|^2$.
Then, $f(z) = (1-|z|^2)^2$, which satisfies $f(1)=f(-1) = 0$, but $f(0) = 1$. Hence, $$f\left(\dfrac{-1+1}{2}\right) = f(0) = 1 > 0 = \dfrac{0+0}{2} = \dfrac{f(-1)+f(1)}{2},$$ and thus, $f$ is not convex.