The way I understand it is that second order stationary processes are characterized by their 1st and 2th moment being time-invariant. That is, $E[Z_{t}] = \tau_{1}$ and $E[Z_{t}^2] = \tau_{2}$, where $\tau_{1}$, $\tau_{2}$ are time-independent. According to Wikipedia, covariance stationary is the same as second-order stationary, so this should imply that $E[Z_{t}Z_{t+k}] = \tau_{3}$, where $\tau_{3}$ is time-independent.
I don't understand why this is the case. Have I misunderstood something? Thanks.
Here are the definitions i'm using:
First-order stationary (i.e. mean stationary): E$[Z_t] = \mu<\infty$ is constant.
Second-moment stationary: E$[Z_t] = \mu<\infty$ and E$[Z_t^2] = \tau<\infty$.
Covariance/weak stationary: E$[Z_t] = \mu<\infty$ and E$[Z_t^2] = \tau<\infty$ and $\text{Cov}[Z_{t+h}, Z_t] = \gamma_h$.
Note that Covariance stationary implies Second-moment stationary: $$ \gamma_0 = \text{Cov}(Z_t, Z_t) = \text{Var}(Z_t) = \mu^2 - E[Z_t^2] $$
But second-moment stationary does not imply Covariance stationary.
Let $X_0,X_1,X_2,\dots$ be i.i.d. with mean $0$ and variance $\sigma^2$. Define $$ Z_t := \dfrac{1}{\sqrt{t+1}} \sum_{k=0}^t X_k. $$ Then E$[Z_t] = 0$ and Var$(Z_t) = \sigma^2$ for all $t$, so E[$Z_t^2] = \sigma^2$ for all $t$. However, \begin{align*} \text{Cov}(Z_{t+h},Z_t) &= \text{Cov} \left( \dfrac{1}{\sqrt{t+h+1}} \sum_{k=0}^{t+h} X_k,\dfrac{1}{\sqrt{t+1}} \sum_{j=0}^t X_j \right) \\ &= \dfrac{1}{\sqrt{t+h+1} \sqrt{t+1}} \sum_{k=0}^{t+h} \sum_{j=0}^t \text{Cov}(X_k,X_j) \\ &= \dfrac{(t+1) \sigma^2}{\sqrt{t+h+1} \sqrt{t+1}}, \end{align*} which clearly depends on $t$.
When I think of second-order stationary (in the context of time-series), I think of weak-stationary i.e. covariance stationary. The following link also might be of use.
https://stats.stackexchange.com/questions/65353/what-is-a-second-order-stationary-process