How to solve second order differential optimal control or optimization problem？

Question

From a long time, I meet an optimal control problem, but i don't know how to solve it. Well, to be more specific,

Suppose we have following dynamic system and cost function,
\begin{cases} \ddot{y}=\dot{y}-rv \\ J_{max}=\int_{0}^{\infty}e^{-\rho t}[(1-\frac{v}{2})v-\sigma y]dt,v\in[0,1] \end{cases} where $\rho$ is a discount factor, $r$ are parameters, $v$ is control, is a random variable, my question is how to solve this kind of question, determine y(t) and control v(t), $\sigma$ is a random variable with prabability discrete distribution function, for explicitly, $\sigma$ is a uniform distribution, when $ 0<\sigma<5 $，f($\sigma$)=$\frac{1}{5}$, else, f($\sigma$)= 0.

if dymamic system is first order differential equation, I know how to solve it, normally it can be use PMP or HJB. Does anyone can be give me some suggestions or comments? Thanks a lot.

Answer 1

Any higher order ordinary differential equation can be turned into a system of first order ordinary differential equations. For example by choosing each next element of the state vector as the derivative of the previous element.

So in your case $\ddot{y}=\dot{y}-rv$ can also be written as \begin{align} \dot{x}_1 &= x_2, \\ \dot{x}_2 &= x_2 - rv, \end{align}

with $x_1=y$ and $x_2=\dot{y}$. After switching to this formulation one can use the normal approach, such as PMP or HJB.

Answer 2

In general, you have two options: Dynamic Programming and Pontryagin Maximum Principle. I would personally go for dynamic programming here since the system is linear and the cost is a quadratic polynomial.

Let $\dot{x}=Ax+Bv$, $y=Cx$ be the system in compact state-space form. Since there is an affine term in the cost, we look for a value function of the form

$$V(x)=x^TPx+2b^Tx+c.$$

The HJB equation is given in this case by (where I am assuming that $\sigma$ is NOT a random variable):

$$-\rho V(x)+\min_v\{V_x(x)(Ax+Bv)+v(1-v/2)\}-\sigma Cx=0.$$

However, the input constraint may be difficult to consider within this framework and the PMP is more suitable when such constraints are involved. There are several version of the PMP available for infinite horizon discounted optimal control problems.