Two problems coming from Optimization in Economics (Permanent Income Hypothesis)
Q1. Why Bellman equation is $$V(s)=r(s)+\beta\sum_{s'} p(s'\mid s)V(s')$$ instead of $$V(s)=r(s)+\beta\sum_{s'} p(s'\mid s)r(s')$$ where $r$ is the payment given state?
Q2. In deterministic case, maximization of $$\sum_{t}\beta^tu(c_t)$$ subject to constraint $$a_{t+1}=\frac{1}{\beta}(a_t+y_t-c_t)$$ gives Euler equation $$\frac{1}{\beta}=\frac{u'(c_{t+1})}{u'(c_t)}$$ by first order condition of Lagrangian
However, in stochastic case, do we have $$\frac{1}{\beta}=\mathbb{E}_t[\frac{u'(c_{t+1})}{u'(c_t)}]$$? Is there actually an stochastic Lagrangian formulation and we set $$\lambda_t$$ depend on $t$ when doing optimization? Intuitively suppose we have income process $$y_{t+1}=y_t+\epsilon_t$$, it's really hard to say how much I will save based on the size of the shock.
If the formulation is hard we can only use the method in Q1
To Q1. $r(s)$ is the payment given that you are at state $s$. But this is only the payment for one period and you have it already in front of the summation. You need the $V(s)$ because you are considering the whole sequence of events that will follow in the future, given that you are now at state $s$. $V(s)$ stands for the whole future after reaching state $s$, including of course future payoffs $r(\cdot)$, while $r(s)$ is only the payoff of the first (current) period.
As an example, consider the case where you have only two states, $s,s'$. Then the Bellman equation is: $$V(s)=r(s)+βp(s\mid s)V(s)+βp(s'\mid s)V(s')$$ and it reads: