In $\mathbb{R}$, let $\psi(x)=e^{-1/x^2}$ for $x>0$ and zero otherwise. If $\mathcal{D}$ is the class of test functions, $\phi\in \mathcal{D}(a<x<b)$ for $\phi = \psi(x)\psi(1-x)$, provided $a<0$ and $b>1$. Why it cannot be that $a\leq 0$ and $b\geq 1$?
$\phi(x)$ has continous derivatives for all orders and vanishes outside $0<x<1$, so I cannot see why $a<0$ and $b>1$ for $\phi$ to be a proper test function.
EDIT: I have the definition: The class of test functions $\mathcal{D}(\Omega)$ consists of all functions $\phi$ defined in $\Omega$, vanishing outside a bounded set $\Omega$ that stays away from the boundary of $\Omega$, and such that all partial derivatives of all orders of $\phi$ are continuous.
So at this point (in principle) I know nothing about support.
The function $\phi$ is defined on $(a,b)$ (if we want it to belong to $\mathcal D((a,b))$). If $0<a<1<b$, then its support is $(a,1]$ which is not compact, and therefore $\phi\notin \mathcal D((a,b))$.
Edit
For a function $f:X\to \mathbb R$ (where $X$ is a topological space), the support of $f$ is the closure of $f^{-1}(\mathbb R\backslash \{0\})$.
Your definition of $\mathcal D((a,b))$ as "staying away from the boundary" is then equivalent to the following : $\phi \in \mathcal D((a,b))$ if, and only if its support is compact (and it is smooth).
Your function $\phi$ is defined on the interval $(a,b)$, therefore its support is : $$\operatorname{supp}(\phi) = (a,b)\cap[0,1]$$
If $[0,1]\subset (a,b)$, then the support is equal to $[0,1]$, and is compact.
If $[0,1]\not \subset (a,b)$, then $\operatorname{supp}(\phi)$ is no longer compact. For example if $0<a<1<b$, then $(a,b)\cap [0,1] = (a,1]$, which is not compact (though it is closed in $(a,b)$).