Let $\{S_n=Y_0+\cdots+Y_n\}_{n\ge 0}$ be a renewal sequence, where $\{Y_n\}_{n\ge 1}$ is a sequence of i.i.d. random variables taking on only non-negative values and assume $Y_0=0$. Let $N(t)=\sum_{n=0}^{\infty}1_{[0,t]}(S_n),$, where $1_{[0,t]}$ is the characteristic function of the interval $[0,t]$. $U(t)=\mathbb{E}[N(t)].$ Call $\mathbb{E}[Y_1]=\mu$. For $t\ge 0$, let $A(t)=t-S_{N(t)-1}$, $B(t)=S_{N(t)}-t$ be backward recurrence time and forward recurrence time, respectively.
The problem is to find $\mathbb{E}[A(t)B(t)]$. And if $E[Y_1^3]<\infty$, find $\lim_{t\to\infty}\mathbb{E}[A(t)B(t)]$.
The following is my solution: since $\mathbb{E}[Y_1^3]<\infty,$ then $\mathbb{E}[Y_1],\mathbb{E}[Y_1^2]<\infty$. Since $A(t)=t-S_{N(t)-1}$, $B(t)=S_{N(t)}-t$, then $A(t)B(t)=tS_{N(t)}-t^2+tS_{N(t)-1}-S_{N(t)-1}S_{N(t)}$, then take expectation of both sides, we get $\mathbb{E}[A(t)B(t)]=t\mathbb{E}S_{N(t)}-t^2+t\mathbb{E}[S_{N(t)-1}]-\mathbb{E}[S_{N(t)-1}S_{N(t)}].$
By Wald's theorem, we know $\mathbb{E}S_{N(t)}=\mathbb{E}[N(t)]\mathbb{E}[Y_1]=U(t)\mu.$ Similarly $\mathbb{E}[S_{N(t)-1}]=(U(t)-1)\mu,$ since $S_{N(t)-1}=S_{N(t)}-Y_{N(t)}$ and take expectation of both sides.
And $S_{N(t)-1}S_{N(t)}=S_{N(t)-1}(S_{N(t)-1}+Y_{N(t)})=S_{N(t)-1}S_{N(t)-1}+S_{N(t)-1}Y_{N(t)}$.
$S_{N(t)-1}S_{N(t)-1}=\sum_{i=1}^{N(t)-1}Y_i\sum_{j=1}^{N(t)-1}Y_j=\sum_{i=1}^{N(t)-1}Y_i\sum_{j=1,j\neq i}^{N(t)-1}Y_j+\sum_{i=1}^{N(t)-1}Y_iY_i.$
Since $S_{N(t)-1}$ and $Y_{N(t)}$ is independent, $Y_i$ and $\sum_{j=1,j\neq i}^{N(t)-1}Y_j$ are independent, then $\mathbb{E}[S_{N(t)-1}S_{N(t)}]=(U(t)-1)(U(t)-2)\mathbb{E}[Y_1]^2+(U(t)-1)\mathbb{E}[Y_1^2]$.
Then $\mathbb{E}[A(t)B(t)]=t(2U(t)-1)\mu-t^2-(U(t)-1)^2\mu^2-(U(t)-1)\mathbb{E}[Y_1^2].$
Since $\lim_{t\to\infty}\frac{U(t)}{t}=\frac{1}{\mu}$, then by calculation $\lim_{t\to\infty}\mathbb{E}[A(t)B(t)]=(1-u(t))Var(Y_1)$.
But since $A(t)B(t)\ge 0$ and $\lim_{t\to\infty}U(t)=+\infty$, we get a contradiction.
I'd like to know where is wrong and how can I solve this problem correctly.
From standard renewal-reward theorems, you can compute a constant $c$ such that:
\begin{align} \lim_{t\rightarrow\infty} \frac{1}{t}\int_0^t E[A(\tau)B(\tau)]d\tau &= c \\ \lim_{t\rightarrow\infty} \frac{1}{t}\int_0^t A(\tau)B(\tau)d\tau &= c \quad \mbox{(with prob 1)} \end{align} Indeed, the constant $c$ involves the third moment of $Y_1$, and is finite whenever $E[Y_1^3]<\infty$. In the special case when the limit of $E[A(t)B(t)]$ exists, it must also be equal to $c$.
However, the limit of $E[A(t)B(t)]$ does not always exist. A counter-example is when $Y_i=1$ for all $i$. Then $A(t)B(t)$ is a deterministic process, so $A(t)B(t)=E[A(t)B(t)]$ for all $t$. In this case, $A(t)B(t)$ is a periodic function that oscillates between zero and nonzero values over its period of size 1.
If you assume additional properties, such as “non-arithmetic” properties of the renewal process, then you can conclude: $$\lim_{t\rightarrow\infty} E[A(t)B(t)] = c$$ For details on the definition of “non-arithmetic,” see, for example, Corollary 3.2 (equation 3.54) in these notes:
http://www.rle.mit.edu/rgallager/documents/Renewal.pdf
As Did mentions, the inspection paradox creates strange dependencies that make some of your calculations incorrect. This includes the calculations where you sum over the $Y_i$ values, but you do not account for dependencies that arise from that fact that the indices $i$ that are used depend on $N(t)$. Your calculation $E[S_{N(t)}]=E[N(t)]\mu$ is correct (using your definition with $N(0)=1$), and so indeed you can compute $E[B(t)]$.
Unfortunately, I have no idea how to compute $E[A(t)B(t)]$ for a particular finite time $t>0$. If you have a way to do that, please let me know.