I have came up with a proof but now sure if the last part is correct/the best way to do it.
Let $f(x),g(x)$ be real-valued continuous functions defined on $[0,\infty)$ satisfying the followings:
(a). $\lim_{x\to\infty} f(x) = 0$
(b). $\int_0^\infty \left| g(x) \right| dx < +\infty$
Prove that if $h(x)$ is defined as
\begin{equation} h(x) = \int_0^x f(x-y) g(y) dy \end{equation}
then $\lim_{x\to\infty} h(x) = 0$.
For convenience, let $\int_0^\infty \left| g(x) \right| dx = M$.
Since $\lim_{x\to\infty} f(x) = 0$, for any $\epsilon > 0$, we can find $K_\epsilon > 0 $ such that whenever $x > K_\epsilon$, $\left| f(x) \right| < \epsilon $ holds. If we want $ x - y > K_\epsilon $, then we need $y < x - K_\epsilon$. This gives us
\begin{align} \left| h(x) \right| &= \left| \int_0^x f(x-y) g(y) dy \right| < \int_0^x \left| f(x-y) \right| \left| g(y) \right| dy\\ & = \int_0^{x - K_\epsilon} \left| f(x-y) \right| \left| g(y) \right| dy + \int_{x - K_\epsilon}^x \left| f(x-y) \right| \left| g(y) \right| dy \\ &< M\epsilon + \int_{x - K_\epsilon}^x \left| f(x-y) \right| \left| g(y) \right| dy \end{align}
When $y\in [x-K_\epsilon ,x]$, we get can a uniform bound for $f(x-y)$ since $f$ is continuous. Let $f(x-y) \le M'$ for all $y\in [x-K_\epsilon, x]$. Then we now have
\begin{equation} \left| h(x) \right| \le M \epsilon + M' \int_{x-K_\epsilon}^x \left| g(y) \right| dy \end{equation}
We claim that as $x\to \infty$, the integral on the right hand side goes to zero. This is because
\begin{align} \left| \int_{x-K_\epsilon}^x \left| g(y) \right| dy \right| &= \left| \int_0^x \left| g(y) \right| dy - \int_0^{x - K_\epsilon} \left| g(y) \right| dy \right|\\ & = \left| \int_0^x g(y) dy - M + M - \int_0^{x - K_\epsilon} g(y) dy \right|\\ &\le \left| \int_0^x g(y) dy - M\right| + \left| \int_0^{x - K_\epsilon} g(y) dy - M\right| \\ & \le \frac{\epsilon}{2} + \frac{\epsilon}{2}=\epsilon \end{align}
and since $\int_0^\infty \left| g(x) \right| dx = M$, we can find $P$ such that when $x > P$, both $\int_0^x \left| g(y) \right| dy $ and $\int_0^{x - K_\epsilon} \left| g(y) \right| dy $ are small.
As a result, $\lim_{x\to\infty} \left| h(x) \right| = 0$.
Any feedback is appreciated!
Another solution using Lebesgue's dominated convergence.
$$|h(x)|\le \int_0^x |f(x-y)g(y)|~dy=\int_0^{\infty}1_{(0,x)}|f(x-y)g(y)|~dy$$
Then note that since $f\to 0$ as $x\to\infty$ and $f$ is continuous, we have that $f$ is bounded by some $M$. So $1_{(0,x)}|f(x-y)g(y)|\le M|g(y)|$ and by assumption $g$ is absolutely integrable. So we may apply dominated convergence. Note for any $y$, $\lim_{x\to\infty} 1_{(0,x)}|f(x-y)g(y)|=0$. The result follows.