I am just starting with MDPs and I have a small question.
Consider an MDP with finite state space $S$, finite action space $A$, transition function $S\times A\to P(S)$ and reward function $g$.
As $S$ and $A$ are finite space, we can endow these space with discrete topology.
We know that every function defined on discrete topology to some other space is continuous by definition.
Therefore, the function $A\to P(s^\prime|s,a)$ is indeed continuous.
However, is the function $S\times A\to P(s^\prime|s,a)$ continuous as well ?
Thank you
Yes. $S \times A$ is also a discrete set, since both $S$ and $A$ are finite. Thus, any function over $S \times A$ is continuous.
Note, however, that since any function on $S \times A$ is continuous, this claim does not provide us with much information.