The control system is as follows : $(\Omega,\mathcal{F},(\mathcal{F}_t),\mathbb{P})$ is a filtered probability space, $dX_t^u = u_tdt + dW_t,\quad t\in[0,T]$ where $u_t$ is a Markov strategy and $W_t$ is an $\mathcal{F}_t$-Wiener process.
The question is to show that $\mathbb{E}[(X_t^u)^2] = \dfrac{1}{2c}-e^{-2cT}\dfrac{1-2c\mathbb{E}[(X_0)^2]}{2c}$ if $u_t = -cX_t^u$ where $c>0$
So I went ahead and did this : By Itô's rule $d\mathbb{E}[(X_t^u)^2] = 2\mathbb{E}[X_t^u](u_tdt + dW_t) + dt = \mathbb{E}[2u_tX_t^u + 1]dt + 2\mathbb{E}[X_t^u]dW_t$
So from my limited understanding, the term $2\mathbb{E}[X_t^u]dW_t$ should equal to zero because $X_t^u$ is independent from $dW_t$ and the expectation between each step of the Wiener process is zero. The thing is, doesn't it work only is we can apply Fubini's theorem? If so, does it suffice to say that these processes are both $\mathcal{F}_t$-measurable?
So once we get to integral form and replace $u_t$, we've got something that looks like this
$\mathbb{E}[(X_t^u)^2] = \mathbb{E}[(X_0)^2] + \displaystyle\int^T_0\mathbb{E}[2u_tX_t^u + 1]dt = \mathbb{E}[(X_0)^2] + \displaystyle\int^T_0\mathbb{E}[-2c(X_t^u)^2 + 1]dt $
And so I'm pretty stuck here because I'm not sure how to proceed with the expectation in this context.
The quick way is to note that you're actually studying an Ornstein-Uhlenbeck process whose explicit solution is $$ X_t^u=X_0 e^{-ct}+\int_0^t e^{-c(t-s)}\,dW_s. $$ Now if $X_0$ is independent of $W$ (otherwise your formula doesn't hold), we have $$ E[(X_t^u)^2]=e^{-2ct}E[(X_0)^2]+2e^{-ct}E[X_0]E\left[\int_0^t e^{-c(t-s)}\,dW_s\right]+E\left[\left(\int_0^t e^{-c(t-s)}\,dW_s\right)^2\right]. $$ The second term vanishes and for the last we have $$ E\left[\left(\int_0^t e^{-c(t-s)}\,dW_s\right)^2\right]=\int_0^t e^{-2c(t-s)}\,ds=\frac{1-e^{-2ct}}{2c} $$ by Ito's isometry. This gives your formula.