In page 188 of the book Adventures in Stochastic Processes, it shows that if $F(dx) = xe^{-x}dx$, then the renewal function $U(x)$ will have the following expression $$U(x) = \frac{3}{4}+\frac{x}{2}+\frac{1}{4}e^{-2x}$$
Now, I want to try it with another approach. That is, I try to find the Laplace transform $\hat{F}$, then find the renewal function $U(x)$ by applying inverse Laplace transform on the renewal equation $\hat{U}(\lambda)=\frac{\hat{F^{0*}}(\lambda)}{1-\hat{F}(\lambda)}$.
My Approach
\begin{aligned} \hat{F} & =\int_0^\infty e^{-\lambda x} (xe^{-x})dx \\ &= \int_0^\infty xe^{-(\lambda+1)x}dx \\ &= \frac{1}{(\lambda+1)^2}\int_0^\infty ye^{-y}dy \ \ \ \ \text{(Let }y=(\lambda+1)x \text{)}\\ &=\frac{1}{(\lambda+1)^2} \end{aligned}
Now, we also know that \begin{aligned} \hat{U}(\lambda)&=\frac{1}{1-\hat{F}} \ \ \ \ \ \text{ (I remember that }\hat{F^{0*}}=1) \\ &=\frac{(\lambda+1)^2}{\lambda^2+2\lambda}\\ &=1+\frac{1}{\lambda^2+2\lambda}\\ &=1+\frac{1}{2\lambda}-\frac{1}{2(\lambda+2)} \end{aligned}
Finally, applying inverse Laplace transform
\begin{aligned} U(x)&=\mathcal{L}^{-1}\{\hat{U}(\lambda)\}\\ &=\mathcal{L}^{-1}\{1+\frac{1}{2\lambda}-\frac{1}{2(\lambda+2)}\}\\ &=\delta(x)+\frac{1}{2}-\frac{1}{2}e^{-2x} \end{aligned}
Which is different from the textbook What's wrong here? Please correct me if I have any misconception
Finally, I discovered that I've made a small mistake on the formula. It should be $$\hat{U}=\frac{\hat{F^{0*}}}{\lambda(1-\hat{F})} \ \ \ \text{ (See Appendix B for the derivation)}$$
After having the correct formula, the steps are easy: \begin{aligned} \hat{U}(\lambda)&=\frac{1}{\lambda(1-\hat{F})} \ \ \ \ \ (\hat{F^{0*}}=1 \text{ by definition}) \\ &=\frac{(\lambda+1)^2}{\lambda(\lambda^2+2\lambda)}\\ &=\frac{3}{4\lambda}+\frac{1}{2\lambda^2}+\frac{1}{4(\lambda+2)} \end{aligned} Now, by using the inverse of Laplace transform, the result will be $$U(x) = \frac{3}{4}+\frac{x}{2}+\frac{1}{4}e^{-2x}$$
Appendix A - Method from the textbook
(I will just type out the exact wordings to prevent any mistakes)
We seek the density of of $\sum_{n=1}^\infty F^{n*}$. Observe that the Laplace transform of $\sum_{n=1}^\infty F^{n*}$ can be written as $$ \begin{align} \Bigg(\widehat{\sum_{n=1}^\infty F^{n*}}\Bigg)(\lambda) &=\sum_{n=1}^\infty \big(\widehat{F^{n*}}\big)(\lambda) = \sum_{n=1}^\infty\big(\hat{F}(\lambda) \big)^n \\ & = \sum_{n=1}^\infty ((1+\lambda)^{-2})^n \\ & = \frac{(1+\lambda)^{-2}}{1-(1+\lambda)^{-2}} = \frac{1}{1+2\lambda+\lambda^2-1} \\ & = \frac{1}{\lambda^2+2\lambda} = \frac{1}{\lambda(\lambda+2)}\\ \text{By a partial fraction expansion, this is}\\ &=\frac{1}{2\lambda}-\frac{1}{2(\lambda+2)} \\ &=\int_{x=0}^\infty e^{-\lambda x}\frac{1}{2}dx-\int_{x=0}^\infty e^{-\lambda x}\frac{1}{2}e^{-2x}dx\\ &=\int_{x=0}^\infty e^{-\lambda x}(\frac{1}{2}-\frac{1}{2}e^{-2x})dx \\ &=\int_{x=0}^\infty e^{-\lambda x}\Bigg(\sum_{n=1}^\infty F^{n*}\Bigg)dx \end{align} $$ Therefore, the required density is $\frac{1}{2}-\frac{1}{2}e^{-2x}$, since the transform uniquely determines the measure. Thus we have $$ \begin{align} U(x)&=\sum_{n=0}^{\infty}F^{n*}(x) \\ &= 1+\int_0^x\frac{1}{2}-\frac{1}{2}e^{-2s}ds\\ &= \frac{3}{4} + \frac{x}{2} +\frac{1}{4}e^{-2x} \end{align} $$
Appendix B - Proof for my formula
$$ \begin{align} U(x)&=\sum_{n=0}^{\infty}F^{n*}(x) \\ &= F^{0*}+ \sum_{n=1}^{\infty}F^{n*}(x) \\ &= 1+ \sum_{n=1}^{\infty}F^{(n-1)*}*F(x) \\ &= 1+ \sum_{n=0}^{\infty}F^{n*}*F(x) \\ &= 1+ U*F(x) \end{align} $$ Now, applies Laplace transform on both side
$$ \begin{align} \ \widehat{U}&=\frac{1}{\lambda}+\widehat{U}\widehat{F} \\ \widehat{U}(1-\widehat{F})&=\frac{1}{\lambda}\\ \widehat{U}&=\frac{1}{\lambda(1-\widehat{F})} \end{align} $$