For context, I currently taking a class on Probability Modeling and I also happen to be teaching Dynamic Programming (Graduate Seminar), in the former class the instructor is starting to teach us about the Markov Chain decision process, and I started to see what I think are connections between both of the topics, as for example, let us envision a Markov Chain diagram like so:
For the Dynamic Problem, letting $t$ denote discrete time with respect to traversing from one node to another, $s_t$ denote the node we are in given stage $t$, and $x_t$ denoting the decision to transfer from $s_t$ to $s_{t+1}$ at time $t$. Then the Dynamic stage diagram from the above Markov Chain diagram would look like the following:
For the Dynamic problem, $t$ has to be finite for the recursive relation to happen at some point. Now that I have done all of this, I am asking myself, "What would this Dynamic process represent with respect to the Markov Chain?"
Well, I came to the notion that given a starting state at time $t=k$, where $k$ is the chosen ending time of the process, the process depicts the optimal min/max from the starting state of $s_k$ all the way back to the states of time $t=0$ with some cost relation like the following:
$$f_t^*(s_t) = \max_{x_t=1,2,3,4} \{f_t(s_t, x_t)\}$$
where,
$$f_t(s_t, x_t) = p(x_t)\cdot f_{t+1}^*(s_t)$$
with $f_k^*=1$ and $p(x_t)$ denoting the probability of traversing from one node to another given the decision at time $t$. In other words, the total probability it would take to get from a node at time $t=0$ to the state we are in at time $t=k$ given we are picking the most optimal route to get there.
Since I am quite the newbie with understanding Markov Chains, I am unsure if I correctly see a connection- here or not, or if I am missing/misunderstanding parts - and if there is a connection between Markov Chains and Dynamic Programming, what are some resources I could look into to learn more?