Let $(E,d)$ be a metric space and $B\subseteq E$ be closed, let $x:[0,\infty)\to E$, $I:=\{t\ge0:x(t)\in B\}$ and $\tau:=\inf I$. If $I$ is nonempty and $\tau\in I$, then we easily see that $$\tau\le t\Leftrightarrow\exists s\in I:s\le t\tag1$$ for all $t\ge0$. Moreover, if $I$ is nonempty and $x$ is right-continuous, then $\tau\in I$.
Now let $(X_t)_{t\ge0}$ be a right-continuous $E$-valued process on a probability space $(\Omega,\mathcal A,\operatorname P)$ and replace $I$ by $I:=\{t\ge0:X_t\in B\}$.
If $\omega\in\Omega$, by the aforementioned facts, either $I(\omega)=\emptyset$ (and hence $\tau(\omega)=\infty$) or $\tau(\omega)\in I(\omega)$ and hence $$\tau(\omega)\le t\Leftrightarrow\exists s\in I(\omega):s\le t\tag2.$$
From $(2)$ we see that $$\{\tau\le t\}=\bigcup_{s\in[0,\:t]}\{X_s\in B\}\tag3$$ for all $t\ge0$.
How can we show that $$\{\tau\le t\}=\bigcup_{\substack{s\in\mathbb Q\\0\le s\le t}}\{X_s\in B\}\tag4$$ for all $t\ge00$?
By $(3)$ we only need to show that if $(\omega,s)\in\Omega\times[0,t]$, there is a $\tilde s\in\mathbb Q$ with $0\le\tilde s\le t$ and $X_{\tilde s}\in B$.
How can we do that? Clearly, if we fix $\varepsilon>0$, we can use that $\mathbb Q$ is dense in $\mathbb R$ to find $\tilde s\in\mathbb Q$ with $|\tilde s-s|<\varepsilon$.
Now we somehow need to use that $X$ is right-continuous and a suitable characterization of closedness of $B$ ...
EDIT
I've found the following proof in Kallenberg:
Our case is item (ii). While he is only claiming that $\tau_B$ is a weakly stopping time (i.e. $\{\tau_B<t\}\in\mathcal F_t$), his proof claims $\{\tau_B\le t\}\in\mathcal F_t$? Is this a typo? Would be somehow strange, since I've taken this from the third edition of the book ...
You need rigth continuity of the filtration for this. Instead of (4) use the following:
$$(\{\tau> t\})=\bigcup_{r\in\mathbb Q,t+\epsilon> r>t}\bigcap_{s<r} (\{X_s \in B^{c})$$ which belongs to $\mathcal F_{t+\epsilon}$.