So I've looked through my book for hours and maybe I'm not googling the right thing but I'm not sure what part b for each of these problems is asking me to do. Can anyone tell me what it means? I know for 4.1.2 it's asking for the probability but I'm not sure of what or in what order.
Let $P\equiv(p_{i,j})$ be the transition matrix associated with this time-homogeneous Markov chain. That is, $$ p_{i,j}\equiv\mathbb{P}(X_{n+1}=j\mid X_{n}=i). $$ For the first question, apply the definition of expectation: $$ \mathbb{E}\left[X_{1}\mid X_{0}=2\right]=\sum_{x}p_{2,x}\cdot x. $$ For the second question, use the Markov property: $$ \mathbb{P}(X_{1}=3,X_{2}=1\mid X_{0}=2)=\mathbb{P}(X_{1}=3\mid X_{0}=2)\mathbb{P}(X_{2}=1\mid X_{1}=3)=p_{2,3}p_{3,1}. $$