I am guessing this is true, because based on the definition of a regular language:
"A language is called a regular language if some finite automaton recognizes it".
Meaning if we have a non empty regular set {0, 1, 11, 01, .....}, then if we remove the substring '0', then the resulting set {1, 11, 01, ....} should still be recognized by the same finite automaton.
Is this correct? Does anyone have a better explanation to this? Thank you! :)
"Recognise" means that it has to figure out what strings are and are not in the language. It's not good enough to just accept all the correct strings; it must reject all the other ones.
However, it is true that if you remove one string from a regular language, the language is still regular, because the set difference between two regular languages is regular. (Or, in other words, regular languages are closed under intersection and complement.) Since every finite language is regular, the closure property is sufficient.
It's reasonably easy to prove the closure property from the fact that regular languages are recognised by finite state automatons. If you have an automaton for a language and you change every accepting state to a non-accepting state and vice versa, you get the automaton for the complement of the language. Also, if you have two languages and an automaton for each, you can construct a new automaton whose states are the cartesian product of the sets of states of the original automata. In the product automaton, if you pick as accepting states all the pairs $<p, q>$ where $p$ and $q$ are both accepting in their respective automata, then you have an automaton for the intersection. (If instead you selected as accepting states all the pairs $<p, q>$ where at least one of $p$ and $q$ is accepting in its respective automaton, then you have the union. And so on for other boolean combinations, including set difference.)