I have a confusion with Theorem 1.4.3 (Mader 1972) of the Diestel book on Graph Theory which states that
Let $0 \ne k \in \mathbb N$. Every graph $G$ with $d(G) \ge 4k$ has a $(k + 1)$-connected subgraph $H$ such that $\epsilon (H) > \epsilon(G) − k$.
and it's proof is as follows
Put $\gamma := \epsilon(G)$ ($\ge2k$), and consider the subgraphs $G' \subseteq G$ such that $$|G'| \ge 2k \text{ and } \|G'\| > \gamma (|G'| − k ). \tag{$*$}$$ Such graphs $G'$ exist since G is one; let $H$ be one of smallest order.
No graph $G'$ as in $(∗)$ can have order exactly $2k$, since this would imply that $\|G'\| > \gamma k \ge 2k^2 > \binom{|G'|}{2}$. The minimality of $H$ therefore implies that $\delta(H) > \gamma$: otherwise we could delete a vertex of degree at most $\gamma$ and obtain a graph $G' \subseteq H$ still satisfying $(∗)$. In particular, we have $|H| \ge \gamma$. Dividing the inequality of $\|H\| > \gamma |H| − \gamma k$ from $(∗)$ by $|H|$ therefore yields $\epsilon(H) > \gamma − k$, as desired.
It remains to show that $H$ is $(k + 1)$-connected. If not, then $H$ has a proper separation $\{U_1,U_2\}$ of order at most $k$; put $H[U_i] =: H_i$. Since any vertex $v \in U_1 \setminus U_2$ has all its $d(v) \ge \delta(H) > \gamma$ neighbours from $H$ in $H_1$ , we have $|H_1| \ge \gamma \ge 2k$. Similarly, $|H_2| \ge 2k$. As by the minimality of $H$ neither $H_1$ nor $H_2$ satisfies $(∗)$, we further have $$\|H_i\| \le \gamma (|H_i| − k)$$ for $i=1,2$. But then \begin{align} \|H|| &\le \|H_1\| + \|H_2\| \\ &\le \gamma (|H_1| + |H_2|- 2k) \\ &\le \gamma (|H| - k) \qquad \text{(as $|H_1 \cap H_2| \le k$),} \\ \end{align} which contradicts $(∗)$ for $H$.
In this proof why they select Graph $G'$ with condition $(*)$?
why we are not able to find a graph $G'$ as in $(*)$ can have order exactly $2k$?
How they would imply the condition $\|G'\| > \gamma k \ge 2k^2 > \binom{|G'|}{2}$ ?
Short answer? Because it works.
Intuitively, if a dense graph has low connectivity, we can split it up into two parts, at least one of which is still fairly dense. This is made formal in the inequality beginning $\|H\| \le \|H_1\| + \|H_2\|$ at the end of the proof.
If we choose a minimum subgraph with some requirement on how dense it is, then there's no such way to split it up, and therefore it cannot have low connectivity.
Once we go through the proof, we learn that the required condition is exactly $(*)$.
Because the inequality your third question is about results in a contradiction. For any graph $G'$, the inequality $\|G'\| \le \binom{|G'|}{2}$ must hold, because $\|G'\|$ is the number of edges in $G$, $\binom{|G'|}{2}$ is the number of pairs $\{v,w\}$ of vertices in $G$, and there's at most one edge between each pair of vertices.
We have $\|G'\| > \gamma k$ from the second part of $(*)$: we have $\|G'\| > \gamma(|G'|-k)$, and we are assuming $|G'|=2k$ here (for the sake of a contradiction), so this actually says $\|G'\| > \gamma(2k-k) = \gamma k$.
We have $\gamma k \ge 2k^2$ because $\gamma \ge 2k$. This holds because we defined $\gamma = \epsilon(G) = \frac12 d(G)$, and the assumption in the hypotheses of the theorem is $d(G) \ge 4k$.
We have $2k^2 > \binom{|G'|}{2}$ because we have assumed (for the sake of a contradiction) that $|G'|=2k$, so $\binom{|G'|}{2} = \binom{2k}{2} = \frac{2k(2k-1)}{2} = 2k^2-k$. And of course $2k^2 > 2k^2 - k$ for all $k>0$.