The following is a question I have had for quite some time related to systems and control. Problems in control theory typically write the state dynamics as $$x_{t+1}= f(x_t,u_t,w_t)$$ where $x_t\in\mathcal{X}$ is the state, $u_t\in\mathcal{U}$ is the control, and $w_t$ is a noise term. This "state space" form is often stated as a general model, which includes problems such as MDPs as a special case.
My question relates to how to specifically instantiate the general model to describe an MDP. The state and action spaces are fine, set $\mathcal{X}$ and $\mathcal{U}$ to be finite sets, but it's not clear how to define the function $f$ and the noise terms $w_t$ to capture the dynamics of an MDP.
Question: Recall that the state in an MDP evolves according to a conditional probability distribution $p(x_{t+1}\mid x_t,u_t)$. What is the function $f$ and the distribution on $w_t$'s such that the general model describes an MDP?
If $\ \cal{X}\ $ and $\ \cal{U}\ $ are the state space and action space of the the Markov decision process you're trying to represent, and its transition probabilities are given by $$ P\big(s_1,s_2,u\big)=Pr\big(x_{t+1}=s_2\,\big|\,x_t=s_1, u_t=u\big)\ , $$ then you can take the noise of your "state space" representation to be a sequence $\ w_1, w_2,\dots,w_t,\dots\ $ of independent random variables taking values in the space $\ \cal{W=X^{X\,\times U}}\ $ of functions $\ w:\cal{X\,\times U\rightarrow X}\ $. If $\ \cal{X}\ $ and $\ \cal{U}\ $ are finite, then so is $\ \cal{W}\ $ and you can take the probability mass function of $\ w_t\ $ to be given by $$ Pr\big(w_t=w\big)=\prod_{x\in{\cal X}\\ u\in{\cal U}}P(x, w(x,u), u)\ . $$ If you now take $\ f:\cal{X\times U\times W\rightarrow X}\ $ to be given by $$ f(x,u,w)=w(x,u)\ $$ the process $\ x_t\ $ generated by $$ x_{t+1}=f\big(x_t,u_t,w_t\big) $$ will be essentially the same stochastic process as that generated by the original Markov decision process.
If $\ \cal{X}\ $ and $\ \cal{U}\ $ are not both finite, then the definition of $\ Pr\ $ given above won't do. However, if $\ \cal{X}\ $ and $\ \cal{U}\ $ are discrete, you can still construct a probability mass functions on the discrete spaces $\ \cal{W_{AB}=X^{A\,\times B}}\ $, where $\ \cal A\ $ and $\ \cal{B}\ $ are finite subsets of $\ \cal{X}\ $ and $\ \cal{U}\ $, by $$ Pr\big(w_t\,_{|\cal A\times B}=w\big)=\prod_{x\in{\cal A}\\ u\in{\cal B}}P(x, w(x,u), u)\ . $$ I expect these can be parlayed by Kolmogorov extension into a probability measure on a $\sigma$-algebra of subsets of $\ \cal{X^{X\,\times U}}\ $, and with this as the distribution of $\ w_t\ $ and the same $\ f\ $ as above, you will be able to get the same result as in the finite case. However, I don't have much current experience in constructions of this sort, so I'm not sufficiently motivated to try and work out all the gory details.