I am interested in finding out more about Markov decision processes where for every state $s = s_{t}$ at time $t$ and action $a = a_t$ applied on $s_t$, there is only one unique resulting state $s_{t+1}$ at time $t+1$. Would it be correct to say then, that the probability of transitioning from state $s_t$ to $s_{t+1}$ given $a_t$ is $P_a(s_t, s_{t+1}) = 1$? In addition that all non-zero elements of the probability/transition matrix that correspond to any two transition states $s_i, s_{i+1}$ given $a_i$ are just equal to $1$?
Is this what it means to have a "deterministic" Markov decision process? I am interested in implementing a numerical solution to a specific problem and would like to read some background and techniques for solutions. I am not sure if it is worth using the conventional techniques/algorithms of MDP or if this "simplified" problem has easier implementations.
Thank you.
Usually, one discusses deterministic Markov processes only if the set of possible states is infinite. If the space of possible states is finite, your deterministic Markov process must eventually reach a cycle (which could be of length $1$. Clearly in that case, it is much more efficient to evolve the state explicitly than to use matrix methods appropriate for non-deterministic finite-states Markov processes.
When the state space is not finite, the situation can be much richer, and in particular, it is not always easy to figure out whether the trajectory will be periodic (a cycle) or will bounce around without repetition. A nice example is the deterministic process that comes from trying to use Newton's method to find the square root of $-1$; depending on the starting guess, the trajectory can end in a fixed point, become a cycle of any desired length, bounce around "pseudo-randomly" without ever repeating, or shoot off to infinity.
Another example with a good deal of complexity is the Collatz sequence; that can certainly be considered a deterministic Markov process on the set of natural numbers, and nobody knows the general nature of all the trajectories.
Still, for the deterministic process, you don't want to use the matrix machinery designed to cope with a general probabilistic Markov process.