Assume we have a graph $G = (V, E)$ with minimal degree $m$. We mark with a probability (independent from the other vertices) of $p$ each vertex. After that we mark each vertex $v$ iff the vertex $v$ and their neighbors are not marked.
So there is a upper bound for the expected value for the number of marked vertices given: $$|V|\left(p + e^{−p(m+1)}\right)$$
And there is also a formula given for the maximal number of marked vertices: $$|V| \frac{1+\ln(m+1)}{m+1}$$
I don't understand how to derive these.
The probability that a vertex is marked in the first round is $p$, and the probability that a vertex of degree $\deg(v)$ is marked in the second round is $(1-p)^{\deg(v)+1}$: the probability that neither that vertex, nor any of its neighbors, were marked in the first round. These are mutually exclusive, so the total probability that a vertex is marked is $$ p + (1-p)^{\deg(v)+1} \le p + (1-p)^{m+1} \le p + e^{-p(m+1)} $$ because $\deg(v) \ge m$ and by using the inequality $1+x \le e^x$. The expected number of marked vertices is the sum, over all vertices, of the probability that they are marked, which is at most $|V|(p + e^{-p(m+1)})$.
You misunderstand the second quantity (or else you are not explaining it correctly). It is not the maximal number of marked vertices. Rather, it's the minimum of the expression above over all choices of $p$. This minimum is achieved by setting $p = \frac{\ln(m+1)}{m+1}$, which gives us $$ p + e^{-p(m+1)} = \frac{\ln(m+1)}{m+1} + e^{-ln(m+1)} = \frac{\ln(m+1)}{m+1} + \frac1{m+1} = \frac{1 + \ln(m+1)}{m+1}. $$ Why do we care about this? Because our goal is to find a small dominating set for our graph. For any $p$, we can perform this two-round procedure, and get an expected number of $|V|(p + e^{-p(m+1)})$ vertices in the dominating set. In particular, for any $p$, there is a dominating set of size at most $|V|(p + e^{-p(m+1)})$. To get the best upper bound on the size of the smallest dominating set, we choose the value of $p$ that minimizes this expression. We conclude that there is a dominating set with at most $$ |V| \cdot \frac{1 + \ln(m+1)}{m+1} $$ vertices.