Let $X$ be a Markov chain with state space $E = {1, 2, 3, 4}$ and transition matrix
$$P = \begin{bmatrix} 0 & 0 & 1 & 0 \\ 0 & 0.4 & 0.6 & 0\\ 0.8 & 0 & 0.2 & 0\\ 0.2 & 0.3 & 0 & 0.5\\ \end{bmatrix} $$
Compute
$$E[f(X_5)f(X_6)|X_4 = 4]$$
for the function $f$ with values $2, 4, 7, $ and $3$ at states $1, 2, 3, $ and $4$ respectively.
My solution:
I assume you can find $E[f(X_5)]E[f(X_6)]$ due to independence.
For $E[f(X_5)]$, since step begins in state $4$ then proceeds one step, I looked at the fourth row and considered all possibilities. State $4$ to state $1$ has probability $0.2$ with value $2$. Likewise, state $4$ to state $2$: $0.3$ with $4$, state $4$ to state $4$: $0.5$ with $3$. Then taking the product of each pair then summing all you get $3.1$.
Then, for $E[f(X_6)]$, you begin in state $4$ but need to take two steps. You can square the transition matrix. You need only consider the fourth row since you know you begin in state $4$. Thus, take row $4$ and multiply by each column, obtaining:
$$P = \begin{bmatrix} 0.1 & 0.27 & 0.38 & 0.25\\ \end{bmatrix} $$
This part seems correct, as the elements sum to $1$.
Then as before take each probability with its corresponding function value in each state, find the products and sum, and I obtained $4.69$.
Finally take the product $(3.1)(4.69) = 14.539$.
The answer should be $14.41$. Where are my errors? And how to obtain the correct solution?
Your independence assumption is not valid.
Hints for a correct approach: $P(X_6=i,X_5=j|X_4=4)=p_{4j}p_{ji}$ from Markov property. Compute this for all possible values of $i$ and $j$ and the use the equation
$E(f(X_6)f(X_5)|X_4=4)=\sum_{i,j} f(i)f(j) P(X_6=i,X_5=j|X_4=4)$.