Let be the following framework :
transition model : There is an initial state noted start and two final states in (4.3) and (4.2). The desired action is performed with a probability of 0.8, but with a probability of 0.2, the agent moves at right angles to the desired direction. If the action of the agent prevails towards a wall (edge of the grid or obstacle in (2,2), the agent remains in his place.
rewards model : the reward is -0.04 for each state except for both terminal states that give +1 or -1.
I want to give the probability of being in each of the states after the sequence after [↑,↑,→] ?
I'm sure I can use a Matrix to compute the probability to have these states rather than doing $p(i,j|[↑,↑,→]),\forall (i,j)\neq(2,2) $
For each action $a$ in $A = \{← , ↑ , → , ↓\}$ you can compute a transition matrix $T_a$ between every state.
If $P_0$ is a vector with zeroes everywhere and 1 for the dimension representing your initial state, the probability distribution after an action $a$ will be $T_a P_0$. Everything is linear, so if you have multiple actions $T_0, T_1,...T_n$ the probability is given by $T_n T_{n-1} ... T_1 T_0 P_0$.
In your case the actions are ↑, then ↑ then → so the probability distribution is given by $T_→ T_↑ T_↑ P_0$.