I was reading the book "Extremal combinatorics" by S.Jukna but I came across a moment which seems confusing to me (probably wrong).
Question: For $v\in V$ the degree $d_G(v)$ is defined as the number of edges incident with $v$, i.e. $d_G(v):=\#\{e\in E: e \ \text{incident with} \ v\}$. In general $d_G(v)$ may not be equal to $|N(v)|$, where $N(v)$ - the set of neighbors of $v$.
The authors says "we have $\binom{d_u}{2}$ possibilities to choose a $2$-element subset of its $d_u$ neighbors." I think that it is not correct. What if $d_u=0,1$? What $d_u=2$ and some edge $e$ is a loop around $u$?
Can anyone explain how to fix that?
The equation
$$\binom{d_u}2= \text{ # ways to select two distinct neighbors of $u$}\tag{$\star$}$$
is correct as long as $G$ is a simple graph, meaning no loops and no multiple edges. The author is implicitly assuming the $G$ is simple, so their argument is fine.
How do I know the author is assuming no loops and multiple edges?
The author must be excluding multiple edges, since the result is blatantly false when you allow them. Just consider the graph with $n=2$ vertices and two parallel edges connecting the two vertices. You can see $2=|E|\le \frac{2}4(1+\sqrt{4\cdot 2-3})\approx 1.618$ is violated. In fact, you can add as many copies of an edge as you like, so you cannot prove any inequality of the from $|E|\le f(|V|)$ for graphs with multiple edges.
The author must be excluding loops, because the result is false when you allow them. Again letting $n=2$, the graph $G=(V,E)=(\{1,2\},\{11,12,22\})$ has no four-cycles, yet $3=|E|\le \frac{2}4(1+\sqrt{4\cdot 2-3})\approx 1.618$ is violated.
I just noticed that you were concerned about what happens when $d_u\in \{0,1\}$. In these cases, $\star$ is still true, since both sides of $(\star)$ are zero. Note that $\binom nk$ is defined as the number of $k$-element subsets of an $n$-element set. Whenever $k>n$, there are $0$ such subsets, so $\binom nk=0$ when $k>n$.