Let $\phi$ be a function defined on an open interval $I=(a,b)$. The closure of the set of points where $\phi\ne 0$ is called the support of $\phi$. If the support of $\phi$ is a compact set, then $\phi$ is called a function of compact support. A test function is an infinitely differentiable function of compact support.
I konw that, when $I=(-\infty, \infty)$, there are a $M$, s.t. for all $|x|>M$, $\phi(x)=0$. But when $I=(a,b)$ and $a, b$ are not $\pm\infty$, I can't figure out the nature of $\phi$. Specially, is $$\lim_{x\to a}\phi(x)=\lim_{x\to b}\phi(x)=0?$$
Thanks.
Depending on your conventions of closure of subsets of an open interval, then either $\phi$ can be anything, or $\phi$ has to be constantly $0$ within some distance of the endpoints.
Most likely you'd mean the closure of $(a, b)$ in $(a, b)$ to be $(a, b)$, and thus not compact. So near the endpoints $\phi$ has to be $0$.