Let's say that we have a grammar with multiple productions. And there is production from B to epsilon.
Is B a useless symbol?
If there is another production, let's say B to aB. Now B is useless, because Ab | epsilon, doesn't lead us into terminal symbol.
Can I now remove B?
When does epsilon belong into context free language?
$\epsilon$ is not useless, by which I mean there are grammars when you really need it. One way of looking at that is that it allows us to say "stop". Consider your example, if we have the grammar with start symbol $B$, terminals $\{a\}$, and rules:
$$B \rightarrow aB | \epsilon$$
This generates the language $\{a, aa, aaa, aaaa \dots\}$. You can't remove the $\epsilon$ production rule, because otherwise the grammar would never be able to get rid of all its nonterminals and generate a string. And you obviously can't remove $B$.
(If $B \rightarrow \epsilon$ is the only thing you can do with a $B$, then yeah, $B$ is pretty useless and it can be gotten rid of.)