We are asked to show that the probability of extinction $\zeta=\lim_{t\to \infty} P\left(X(t)=0\right)$ given by: $$\zeta=\begin{cases}1&\text{if }\lambda\le \mu,\\ \left(\frac \mu\lambda \right)^{i}&\text{if }\lambda\gt\mu.\end{cases}$$
where $i$ is the initial population size.
Now I found the probability generating function of the BDP: $$P(t,s)=\left({{r\mu-1\over r\lambda-1}}\right)^{i}$$ Where $r=e^{(\lambda-\mu)t}\left({{1-s\over \mu-\lambda s}}\right)$
But afterwards I get stuck when trying to find $P\left(X(t)=0\right)=p_{0}(t)$
Any help is greatly appreciated!
Here is a solution entirely based on a dynamical system approach and on the basic property of branching processes that the process started at time $0$ dies at or before time $t+s$ if and only if each of its subprocesses existing at time $s$ dies at or before time $t$.
Hence, let $\vartheta_t$ denote the probability that the process starting from one particle dies at or before time $t$. At time $s\ll1$, there are zero particle with probability $\mu s+o(s)$, one particle with probability $1-(\lambda+\mu)s+o(s)$, two particles with probability $\lambda s+o(s)$, and more than two particles with probability $o(s)$. Hence, $$ \vartheta_{t+s}=[\mu s+o(s)]+[(1-(\lambda+\mu)s+o(s))\vartheta_t]+[(\lambda s+o(s))\vartheta_t^2]+[o(s)]. $$ Reordering and considering the limit $s\to0$, one sees that $(\vartheta_t)_t$ solves the dynamics $\vartheta_0=0$ and $$ \dot\vartheta_t=U(\vartheta_t),\qquad U(x)=\lambda x^2+\mu-(\lambda+\mu)x. $$ Note that $U(0)=\mu\gt0$ and $U(1)=0$. Furthermore:
Thus $q=\min(1,\mu/\lambda)$ is the probability that the process starting from one particle dies. Starting from $i$ particles, the process dies if and only if each subtree posterior to an initial particle dies. These events are independent hence the whole tree dies with probability $q^i$.