I want to find the covariance of the following SDE:
$$x_{t} = x_{0}e^{-\alpha t} + \rho \int_{0}^{t} e^{-\alpha(t-s)}dW_{s}$$
To start, I find the mean, it is simply:
$$ E[x_{t}] = E[x_{0}] e^{-\alpha t} $$
Then I need to compute:
$$ Cov(x_{t}, x_{s}) = E \left[ (x_{t} - E[x_{t}]) (x_{s} - E[x_{s}]) \right] $$
Expanding this out, gives me:
$$ Cov(x_{t}, x_{s}) = E\left[ (x_{0}e^{-\alpha t} + \rho \int_{0}^{t} e^{-\alpha(t-k)}dW_{k} - E[x_{0}] e^{-\alpha t}) (x_{0}e^{-\alpha s} + \rho \int_{0}^{s} e^{-\alpha(s-q)}dW_{q} - E[x_{0}] e^{-\alpha s}) \right] $$
$$ Cov(x_{t}, x_{s}) = E\left[ x_{0}^{2} e^{-\alpha(t+s) } + x_{0} \rho\int_{0}^{s} e^{-\alpha (s-q)} dW_{q} - x_{0}E[x_{0}] e^{-\alpha(t+s)} + a_{0}\rho e^{-\alpha s} \int_{0}^{t} e^{-\alpha(t-k)} dW_{k} + \rho^{2} \int_{0}^{t} \int_{0}^{s} e^{-\alpha (t-k)} e^{-\alpha (s-q)} dW_{k} dW_{q} - \rho E[x_{0}] e^{-\alpha s} \int_{0}^{t} e^{-\alpha(t-k)} dW_{k} - x_{0}E[x_{0}] e^{-\alpha (t+s)} - \rho E[x_{0}] e^{-\alpha t} \int_{0}^{s} e^{-\alpha (s-q)} dW_{q} + E[x_{0}]^{2} e^{-\alpha(t+s)} \right]$$
Since $E\left[\rho \int_{0}^{t} e^{-\alpha(t-k)}dW_{k} \right] = 0$, we have:
$$ Cov(x_{t}, x_{s}) = \left(E[x_{0}^{2}] - E[x_{0}]^{2}\right)e^{-\alpha (t+s)} + \rho^{2} E\left[\int_{0}^{t} \int_{0}^{s} e^{-\alpha (t-k)} e^{-\alpha (s-q)} dW_{k} dW_{q} \right]$$
This is where I get stuck.
Question: How can I compute $E\left[\int_{0}^{t} \int_{0}^{s} e^{-\alpha (t-k)} e^{-\alpha (s-q)} dW_{k} dW_{q} \right]$?
It doesn't seem to follow the Wiener isometry rule.
Any ideas?
Itô's isometry shows that
$$\mathbb{E} \left( \left[ \int_0^t f(s) \, dW_s \right]^2 \right) = \mathbb{E} \left( \int_0^t f(s)^2 \, ds \right).$$
Using the polarization identity $$x \cdot y = \frac{1}{4} ((x+y)^2-(x-y)^2),$$ this gives
$$\mathbb{E} \left( \int_0^t f(s) \, dW_s \int_0^t g(r) \, dW_r \right) = \mathbb{E} \left( \int_0^t f(s) g(s) \, ds \right). \tag{1}$$
$(1)$ allows you to compute the expectation of the product of stochastic integrals, which appears in the computation of the covariance.
Proof of (1): $$\begin{align*} & \mathbb{E} \left[ \left( \int_0^t f(s) \, dW_s \right) \cdot \left( \int_0^t g(r) \, dW_r \right) \right]\\ &= \frac{1}{4} \left( \mathbb{E} \left[ \left( \int_0^t f(s)\, dW_s + \int_0^t g(r) \, dW_r \right)^2 \right] - \mathbb{E} \left[ \left( \int_0^t f(s) \, dW_s - \int_0^t g(r) \, dW_r \right)^2 \right] \right) \\ &= \frac{1}{4} \left( \mathbb{E} \left[ \left( \int_0^t f(s)+g(s) dW_s \right)^2 \right] - \mathbb{E} \left[ \left( \int_0^t f(s)-g(s) \, dW_s \right)^2 \right] \right) \\ &= \frac{1}{4} \left( \mathbb{E} \left( \int_0^t (f(s)+g(s))^2 \, ds \right) - \mathbb{E} \left( \int_0^t (f(s)-g(s))^2 \, ds \right)\right) \\ &= \mathbb{E} \left[ \left( \int_0^t f(s) \cdot g(s) \, ds \right) \right] \end{align*}$$