Consider a continuous time Markov process $\{X(t)\}_{t≥0}$ on the state space $\{0, 1, 2, . . .\}$ with stationary probabilities $\{π_0, π_1, π_2, . . .\}$. Suppose that, when currently in state $i$, the process will jump to state $j$ after an exponential amount of time with rate $q_{ij}$ and that all exponential times are independent. Assume that $X(0) = 0.$
(a) Let ν be the rate of departure from state $0$. Write ν in terms of the $q_{ij}$ .
(b) Let $Y$ be the time of the first exit from state zero. Find the distribution of $Y$.
(c) Starting from $0$, let R be the time of the first return to state zero. What is E[R]?
(d) Let T be the first time that the process has been in state $0$ for at least τ units (continuous) of time. Show that $$ E[T] = \dfrac{1}{\pi_0 \nu}(e^{\nu \tau}-1) $$ (Hint: Condition on the time of the first exit from state $0$.)
I can gather easily the answers for $a-c$ :
(a) $\nu = \Sigma_{i=1}^\infty q_{0i}$
(b) $Y $ ~ $ \text{Exp}(\nu)$
(c) $E[R]=\dfrac{1}{\nu \pi_0}$
However, I'm having trouble with part (d), as I believe every time we are at state 0, then we should be there for longer than $\tau$ units with probability $e^{-\nu \tau}$ since $P(Y>\tau)=e^{-\nu \tau}$, so it seems like T ~ Geometric($e^{-\nu \tau}$), in which case $E[T]=e^{\nu \tau}$, which is clearly not the answer given. I am also having trouble understanding how to understand the hint. What is the flaw in my thinking? Thank you
Indeed, one should condition on $Y$. If $Y\geqslant\tau$, then $T=\tau$. Otherwise, $T=Y+S+T'$ where $S$ denotes the time needed to come back to $0$ after one leaves $0$ and $T'$ is distributed as $T$. Thus, $$E(T)=\tau P(Y\geqslant\tau)+E(Y;Y\lt\tau)+E(S;Y\lt\tau)+E(T';Y\lt\tau).$$ By independence, $E(S;Y\lt\tau)=E(S)P(Y\lt\tau)$ and $E(T';Y\lt\tau)=E(T')P(Y\lt\tau)$. By definition of $T'$, $E(T')=E(T)$. By definition of $S$, $R=Y+S$ hence $E(S)=E(R)-E(Y)$. Thus, $$E(T)=\tau P(Y\geqslant\tau)+E(Y;Y\lt\tau)+E(R)P(Y\lt\tau)-E(Y)P(Y\lt\tau)+E(T)P(Y\lt\tau).$$ For every exponentially distributed $Y$, one has $$\tau P(Y\geqslant\tau)+E(Y;Y\lt\tau)=E(Y)P(Y\lt\tau),$$ hence one gets $$E(T)=\frac{E(R)P(Y\lt\tau)}{P(Y\geqslant\tau)}.$$ Using once again the distribution of $Y$, and the identity $$E(R)=\frac1{\nu\pi_0},$$ one gets the formula given as the solution to (d). (This solution uses the answers to (b) and (c). Note that the answer to (c) in the post lacks a factor $\nu$, which have been added since this answer was posted.)