In solving a linear stochastic differential equation, I ended up on this integral:
$$\int_0^t\int_0^s e^{-\lambda(s-u)} dW_u ds $$
where $W_u$ is a Wiener process. I've tried solving the integral by writing it as:
$$\int_0^te^{-\lambda s}\left(\int_0^s e^{\lambda u} dW_u\right) ds $$
and integrating by parts by computing the derivative of $\int_0^s e^{\lambda u} dW_u$. However, this would require a sort of fundamental theorem of calculus for stochastic integrals, but I'm not sure it holds.
If I am not mistaken there are two approaches that lead to two different expressions for the double integral (which are of course equal):
You can apply stochastic Fubini: \begin{align} I_t&:=\int_0^t\int_0^s e^{-\lambda(s-u)}\,dW_u\,ds =\int_0^t\int_u^t e^{-\lambda(s-u)}\,ds\, dW_u\\ &=\int_0^t\frac{1-e^{-\lambda(t-u)}}{\lambda}\, dW_u \tag{1} =\frac{W_t-\int_0^te^{-\lambda(t-u)}\,dW_u}{\lambda}\,. \end{align}
Alternatively, if you set
$$ A_u=\frac{1-e^{-\lambda(t-u)}}{\lambda} $$ you can use the integration by parts formula \begin{align}\tag{2} A_tW_t=\int_0^tA_u\,dW_u+\int_0^tW_u\,dA_u\,. \end{align} (The quadratic covariaton term $\langle A,M\rangle_t$ that is usually added here vanishes because $A_t$ has finite variation.) Since $$ dA_u=-e^{-\lambda(t-u)}\,du\,,\quad A_t=0\,, $$ we see that (2) becomes $$ 0=I_t-\int_0^tW_u e^{-\lambda(t-u)}\,du\,. $$ Summary: The double integral can be written in the following alternative ways $$ \boxed{I_t=\int_0^t\frac{1-e^{-\lambda(t-u)}}{\lambda}\, dW_u =\frac{W_t-\int_0^te^{-\lambda(t-u)}\,dW_u}{\lambda} } $$ or $$ \boxed{I_t=\int_0^tW_u e^{-\lambda(t-u)}\,du\,.} $$