Let $(\rho_t)_{t\in[0,\infty]}$ be a family of positive probability densities on $\mathbb R^d$, namely $\rho_t>0$ and $$ \int_{\mathbb R^d} \rho_t(x)\, dx =1 \;.$$ Suppose that: $$ \int_{\mathbb R^d}\rho_t(x)\,|\log\rho_s(x)|\,d x \,<\infty \quad\textrm{for every }t\geq0\textrm{ and }s=0,t,\infty \;;$$
$$\|\rho_t-\rho_0\|_{TV} \,:=\, \frac{1}{2}\,\int_{\mathbb R^d}|\rho_t(x)-\rho_0(x)|\,d x \,\to\,0 \quad\textrm{as } t\to0\;;$$
$$D_{KL}(\rho_t\|\rho_\infty) \,:=\, \int_{\mathbb R^d}\rho_t(x)\,\log\frac{\rho_t(x)}{\rho_\infty(x)}\,d x \,\to D_0\,\in[0,\infty)\quad\textrm{as } t\to0\;.$$
Can we conclude that:
$$ D_{KL}(\rho_0\|\rho_\infty)\,=\, D_0 \quad?$$
No, I don't think so. For notational convenience, I will work with discrete measures; you can convolve them with square pulses to get something continuous. Consider $p_\infty[i] \propto \alpha^{-|i|}$
Then, the relative entropy $D(p || p_\infty)$ for $p$ reduces to:
$\sum_{i \in \mathbb{Z}} p_i \log p_i + |i| p_i \log \alpha - \log 2(1 - \frac{1}{\alpha})$
Now, let $p_{\frac{1}{n}} = \frac{1}{n}$ on $n$, $\frac{1}{n}$ on $-n$, and $0$ otherwise. Then, note that the relative entropy converges to $-\log 2(1 - \frac{1}{\alpha}) + \log \alpha$, but $D(p_0 || p_\infty) = -\log 2(1 - \frac{1}{\alpha})$