I understand the simple case where
$X \to Y \to Z$ form a Markov Chain if their joint distribution is $P(x,y,z) = P(x)P(y\mid x)P(z\mid y)$
However, I have trouble understanding the statement below.
When the joint distribution $p(x_1, x_2, y_1, y_2) = p(x_1,x_2)p(y_1\mid x_1) p(y_2\mid x_2)$, $Y_1 \to X_1 \to X_2 \to Y_2$ forms a Markov Chain
I can't see why Markov Chain is in that order. What am I missing?
Hoping I understood the question right, you must prove that the chain you wrote is a Markov chain. Than, if you change the order (for example $Y_2$ before everything else), you may see that you don't find a Markov chain.
All you have to do is verify the Markov property at each stage. At $X_1$ it is easy. At $X_2$ you have $P(X_2 = x_2 | X_1 = x_1, Y_1 = y_1) = \frac{P(X_2 = x_2, X_1= x_1, Y_1 = y_1)}{P(X_1 = x_1, Y_1 = y_1)}$. Now you have to find the marginal probability distributions, which is easy because you have the joint probability distribution. You find that $P(X_1 = x_1, Y_1 = y_1)= p(x_1)p(y_1\mid x_1)$ and $P(X_2 = x_2, X_1= x_1, Y_1 = y_1)= p(x_1,x_2)p(y_1\mid x_1)$. So you find $P(X_2 = x_2 | X_1 = x_1, Y_1 = y_1) = P(X_2 = x_2 | X_1 = x_1)$, which is the Markov chain property. The same you can do in every other step.
I hope this is of help.