Given that $X(t)$ is an Ornstein-Uhlenbeck process with $X(0) = x_0$, which is a Markov process, but not a Martingale, how could I go forward if I would like to calculate
$E[\int_0^T X(s)ds | \mathcal{F}_t]$?
I have been twisting my brain for hours, but can't seem to find any reasonable approach.
I will try to reply to your question with some assumptions as you were not precise in your description.
Let $(\Omega,\mathcal{F},\lbrace{\mathcal{F_t}\rbrace}_{t\in\mathbb{R}_+},P)$ be a filtered probability space where we defined a $\lbrace{\mathcal{F_t}\rbrace}-$brownian motion $\lbrace{W_t\rbrace}_{t\in\mathbb{R}_+}$ starting from 0. We suppose that the process $X$ satisfies the following SDE: \begin{equation} dX_t = \theta(\mu-X_t)dt + \sigma dW_t \end{equation} where $\sigma >0, \theta >0, \mu \in \mathbb{R}$ and $X_0 \in L^2$ and independant to $W$. We can show that it exists a unique strong solution to this SDE and is given by: \begin{equation*} \forall t\in \mathbb{R}_+, \quad X_t = e^{-\theta t}X_0 + \mu(1+e^{-\theta t}) +\sigma\int_0^te^{-\theta(t-s)}dW_t \end{equation*} The problem was to compute: for $T>t$, $E\left[\int_0^TX_sds|\mathcal{F_t}\right]$ \begin{align*} E\left[\int_0^TX_sds|\mathcal{F_t}\right] &= E\left[\int_0^T\left(e^{-\theta s}X_0 + \mu(1+e^{-\theta s}) +\sigma\int_0^se^{-\theta(s-u)}dW_u\right)ds|\mathcal{F}_t\right] \\ &=\int_0^T\left(e^{-\theta s}X_0 + \mu(1+e^{-\theta s})\right)ds + \sigma E\left[\int_0^T\int_0^se^{-\theta(s-u)}dW_uds|\mathcal{F}_t\right] \quad \\ &= \int_0^T\left(e^{-\theta s}X_0 + \mu(1+e^{-\theta s})\right)ds + \sigma E\left[\int_0^T\int_u^Te^{-\theta(s-u)}dsdW_u|\mathcal{F}_t\right] \text{Fubini stochastic} \\ &= \int_0^T\left(e^{-\theta s}X_0 + \mu(1+e^{-\theta s})\right)ds + \sigma E\left[\int_0^T\phi(u)dW_u|\mathcal{F}_t\right] \text{where} \ \phi(u) := \int_u^Te^{-\theta(s-u)}ds \\ &= \int_0^T\left(e^{-\theta s}X_0 + \mu(1+e^{-\theta s})\right)ds + \sigma \int_0^t\phi(u)dW_u \quad \text{a.s.} \end{align*} The last equality holds because the function $\phi$ is $L^2(\mathbb{R}_+, dt)$. Thus the stochastic integral is a Wiener integral (which is a martingale).