I'm having difficulty understanding the proof that allows us to go from the Hamilton-Jacobi-Bellman equation to to the Pontryagin Min(Max) Principle.
Lets consider $x(t)$ and $u(t)$ as real valued functions of time (not vectors)
Notation, let $dx/dt = g(x,u,t)$ and the objective function be $\int_0^T f(x,u,t) dt$, which we wish to minimize with respect to $u(t)$ and $V$ be the value function $V(x,t)= \min_{u} \left\{ \int_t^T f(x,u,\tau)\,d\tau \right\}$
Starting with HBJ we have
\begin{equation} \dfrac{\partial V(x,t)}{\partial t}= - \min_{u} \left\{f(x,u,t) + \dfrac{\partial V(x,t)}{\partial x} g(x,u,t)\, \right\} \end{equation}
Now the standard argument goes if you define the adjoint variable
$p(t)=\partial V(x^*,t)/\partial x$
where $x^*$ is the state associated with the optimal choice of $u$, $u^*$, and define the Hamiltonian
$H(x,u,p,t) \equiv F(x,u,t) + p(t) f(x,u,t)$
Differentiating both sides of the HJB equation with respect to $x$ gives
$\dfrac{dp}{dt} = - \dfrac{\partial H(x^*,u^*,p,t)}{\partial x}$
My concern is that I don't really understand what differentiating both sides with respect to $x$ means here. Now if we ignored the min symbol and ignored all the star symbols, took the derivatives and then plugged the stars back in at the end, then yes it is clear to me that taking the derivative of both sides with respect to $x$ yields the adjoint equation (but this can't be what is going on, without some additional subtle argument ... at least I think). But it is unclear to me how the derivative with respect to $x$ moves through the min. Below I will highlight my ignorance. Say you plugged in the stars into the HJB equations above, i.e. we have
\begin{equation} \dfrac{\partial V(x^*,t)}{\partial t}= - \left[f(x^*,u^*,t) + \dfrac{\partial V(x^*,t)}{\partial x} g(x^*,u^*,t)\, \right] \end{equation}
OK above I plugged in an $x^*$ in on the left but I don't really quite understand that. But going with that
\begin{equation} \dfrac{\partial}{\partial t}\dfrac{\partial}{\partial x} [V(x^*,t)]= - \dfrac{\partial}{\partial x} \left[H(x^*,u^*,p,t) \right] \end{equation}
So I guess one way in which I am confused is that there are already $x^*$ plugged into these functions, doesn't that mean there is no $x$ in them anymore to differentiate with respect to x with
Also on the right hand side is it that
\begin{equation} \dfrac{\partial}{\partial x} \left[H(x^*,u^*,p,t) \right]=\dfrac{\partial H(x^*,u^*,p,t)}{\partial x} + \dfrac{\partial H(x^*,u^*,p,t)}{\partial u}\dfrac{\partial u}{\partial x} \end{equation}
and the term on the right goes away because of $\partial H/\partial u=0$. Am I just botching up multivariable calculus? I'm really having trouble thinking about the stars and how this affects the derivatives we are taking. I guess my question kind of boils down to ... it kind of feels that what is going on here is saying if the first derivative at a point is some number then the second derivative is too (which is generally false). So Know my logic is quite wrong while thinking about this standard proof.
In pretty much any think I've read they never actually walk the reader through this differentiation, I'm wondering if someone can do that here.
OK, so I am not sure if this is right but here is what I think is the answer. Indeed I believe I had my multivariable calculus backwards. Starting with the HJB we have
\begin{equation} \dfrac{\partial V(x,t)}{\partial t}= - \min_{u} \left\{f(x,u,t) + \dfrac{\partial V(x,t)}{\partial x} g(x,u,t)\, \right\} \end{equation}
now let $u^*$ be the optimal solution. The above leads to
\begin{equation} \dfrac{\partial V(x,t)}{\partial t}= - f(x,u^*,t) + \dfrac{\partial V(x,t)}{\partial x} g(x,u^*,t) \end{equation}
Now differentiating both sides with respect to $x$ yields
\begin{equation} \dfrac{\partial}{\partial x} \dfrac{\partial V(x,t)}{\partial t} = - \dfrac{\partial}{\partial x}f(x,u^*,t) - \dfrac{\partial}{\partial x}\left[\dfrac{\partial V(x,t)}{\partial x} g(x,u^*,t)\right] \end{equation} One must now perform the product rule on the term on the right, because both terms in the brackets are functions of $x$
\begin{equation} \dfrac{\partial^2 V(x,t)}{\partial x\partial t} = - \dfrac{\partial}{\partial x}f(x,u^*,t) - \dfrac{\partial^2 V(x,t)}{\partial x^2} g(x,u^*,t) - \dfrac{\partial V(x,t)}{\partial x}\dfrac{\partial g(x,u^*,t)}{\partial x} \end{equation}
Now here is the key step that I was really missing, $dp/dt$ is a total derivative with respect to time, not a partial derivative. Observe that the total derivative of $\partial V / \partial x$ with respect to $t$ is
\begin{align} \dfrac{d}{dt}\left[ \dfrac{\partial V(x,t)}{\partial x} \right] &= \dfrac{\partial }{\partial x}\left[ \dfrac{\partial V(x,t)}{\partial x} \right]\dfrac{dx}{dt} + \dfrac{\partial^2 V(x,t)}{\partial x\partial t}\\ &= \dfrac{\partial^2 V(x,t)}{\partial x^2} g(x,u^*,t) + \dfrac{\partial^2 V(x,t)}{\partial x\partial t} \end{align}
Now if we substitute this into the equation two lines above we get
\begin{equation} \dfrac{d}{dt}\left[ \dfrac{\partial V(x,t)}{\partial x} \right] = - \dfrac{\partial}{\partial x}f(x,u^*,t) - \dfrac{\partial V(x,t)}{\partial x}\dfrac{\partial g(x,u^*,t)}{\partial x} \end{equation}
Now noting that this is true for any $x$ that solves the HJB, it must also be true for the optimal x.
\begin{equation} \dfrac{d}{dt}\left[ \dfrac{\partial V(x^*,t)}{\partial x} \right] = - \dfrac{\partial f(x^*,u^*,t)}{\partial x} - \dfrac{\partial V(x^*,t)}{\partial x}\dfrac{\partial g(x^*,u^*,t)}{\partial x} \end{equation}
which yields the classical result of the max principle
\begin{equation} \dfrac{dp(t)}{dt} = - \dfrac{\partial f(x^*,u^*,t)}{\partial x} - p(t) \dfrac{\partial g(x^*,u^*,t)}{\partial x} \end{equation}
Somewhat magically the $p(t)$ no longer depends on $x$ as it did in the above calculation, so the right hand side is infact the partial derivative of the Hamiltonian with respect to $x$.
So we don't just differentiate both sides with respect to $x$ as most papers/books suggest. We differentiate both sides with respect to $x$ and then must notice that a combination of terms is in fact the total derivative we want. The books just assume an astute reader will notice this. Few, ... a good reminder on the importance of keeping your total and partial derivatives straight! Wow I feel silly now. Well at least I think this is right, perhaps it isn't. If not leave a comment so I can fix it, or post your own answer.