Condition for a Markov process.

Question

In one of my questions on PSE there is a debate in the comments about whether: $$P(n,t+\Delta t)=\sum_m^M P(m,t)P(n,t+\Delta t\mid m,t)\tag{$\star$}$$ holds for general stochastic processes or just for a Markovian processes. Given the vote distribution on the comments several people appear to think that it does only hold for a Markovian. Please can someone explain to me if this or isn't the case?

Answer 1

I am not 100% sure on your notation, but I'm pretty sure it means $$ P(n,t) = \mathbb{P}(X_t = n) \quad\text{and}\quad P(n,t+\Delta t \mid m,t) = \mathbb{P}(X_{t+\Delta t} = n \mid X_t = m)?$$ It's quite confusing to use $n$ and $m$ for the location, since it's often used for a discrete time process! As such, I'm going to use $i$ and $j$, just to be completely clear. Also, it isn't at all clear what your sum is over; I assume over the whole space?

Given this, it holds for any process. Indeed, let's derive the result, using only basic probabilistic results (law of total probability/expectation/Bayes' theorem, whatever you want to call it!): $$ \mathbb{P}(X_{t+s} = i) = \sum_j \mathbb{P}(X_{t+s} = i \mid X_t = j) \mathbb{P}(X_t = j). \tag{$\star$} $$ I've literally done almost nothing here. I've just used that $$ \mathbb{P}(A) = \sum_j \mathbb{P}(A \mid B_j) \mathbb{P}(B_j) $$ where $\{B_j\}_j$ partitions the space (ie 'gives all the options'). Assuming my interpretation of your notation, my $(\star)$ is exactly the same as yours.

I don't know what the other terms in your linked question mean, so I can't help with that unfortunately. The interesting part about a Markovian process is that they usually are time-homogeneous, in the sense that $$ \mathbb{P}(X_{t+s} = i \mid X_t = j) = \mathbb{P}(X_s = i \mid X_0 = j). $$ (There is theory on time-inhomogeneous Markov chains, but this these are studied much less, for better or worse.) Note that this condition does not require the chain to be Markovian. However, how would one usually prove this property? -- by using the Markov property.

Hence, for small $s = \Delta t$, you can 'not worry about the past'. Without really understanding what the notation means, I think that in your linked question the second bullet point implicitly uses this, and hence is only valid for a Markovian process.