Hamilton Jacobi Bellman - boundary conditions

965 Views Asked by At

Consider the optimal control problem where we minimize a Lagrangian $L$: $$ \min \int_0^T L(x(s),u(s),s) ds, \text{ T being fixed} $$ under the differential constraint $$ \dot x = f(x,u),\; u\in U $$

Take the Hamilton Jacobi Bellman (HJB) equation

$$ \frac{\partial \varphi}{\partial t} + \min_u \{L(x,u,t) + \nabla_x\varphi\cdot f(x,u) \}=0 $$ with boudary conditions $$ \varphi(T,x) = 0 $$

Now I consider $T$ as a free variable, the HJB equation becomes

$$ \min_u \{L(x,u,t) + \nabla_x\varphi\cdot f(x,u) \}=0 $$

without boundary conditions on the solution $\varphi$ ?

To reformulate, what is the boundary condition for the solution $\varphi$ of the HJB when the horizon time is free ?

1

There are 1 best solutions below

0
On

In the first case the boundary condition $\varphi(T,x) = 0$ came from fact that your terminal cost was zero. If your terminal cost was $K(x_f)$ or in other words if the cost function was,

$$J\left( u \right): = \int\limits_0^T {L\left( {x\left( s \right),u\left( s \right),s} \right)ds} + K\left( {{x_f}} \right)$$

we would have $\varphi \left( {T,{x_f}} \right) = K\left( {{x_f}} \right)$. Since you had zero terminal cost, there was the zero boundary condition. In general in infinite horizon problem, the boundary condition is zero since as you showed, the PDE became an ODE. On the other hand there is now a bigger problem: Now we have

$$ \int_0^ {\infty} L(x(s),u(s),s) ds $$

First of all, does the integral converge? We possibly don't know without further structure to the problem and to the nature of $L(x,u,t)$ like monotone/non-increasing etc. If it does not then the optimal cost might be infinite and then the value function also is infinite. I think that is a bigger problem than looking for the boundary condition. In certain cases, LQR, discounted costs, positive/negative programming we can indeed establish that the integral converges and find the optimal control.