From paper (page 2 ) we have \begin{eqnarray} r\left( AB\pm BA\right) &\leq &\left\Vert AB\right\Vert +\min \left\{ \left\Vert A\right\Vert ^{1/2}\left\Vert AB^{2}\right\Vert ^{1/2},\left\Vert A^{2}B\right\Vert ^{1/2}\left\Vert B\right\Vert ^{1/2}\right\} \label{e.0.9} \\ &\leq &\left\Vert AB\right\Vert +\left\{ \begin{array}{ll} \left\Vert A\right\Vert ^{1/2}\left\Vert B\right\Vert ^{1/2}\left\Vert AB\right\Vert ^{1/2}, & \\ & \\ \min \left\{ \left\Vert A\right\Vert \left\Vert B^{2}\right\Vert ^{1/2},\left\Vert A^{2}\right\Vert ^{1/2}\left\Vert B\right\Vert \right\} , & \end{array}% \right. \notag \end{eqnarray}
I don't understand what the author mean by the second inequality and why it holds

The passage you quote is a direct reference to
F. Kittaneh, Spectral radius inequalities for Hilbert space operators, Proc. Amer. Math. Soc. 134 (2006), 385-390
which I presume you looked at to find an explanation for the bizarre syntax in the quoted passage.
The first line in your quotation is Kittaneh's equation 13 (in Cor. 3), after applying monotonicity of the square root to exchange the order of roots and minimization. The next line is a list of two more quantities no less than the right-hand side of the first line, obtained from the first line by properties of min and norm.
The quoted inequalities are of the form \begin{align*} r(U) &\leq V \\ &\leq ||A B|| + \begin{cases} X , \\ Y, \end{cases} \text{.} \end{align*} The intended meaning is \begin{align*} r(U) &\leq V \text{,} \\ V &\leq ||AB|| + X \text{, and} \\ V &\leq ||AB|| + Y \text{.} \end{align*}