I want to prove the following:
Let $m^*(E) < \infty$. Show that if $E$ is not measurable,then there is an open set $\mathcal{O}$ containing $E$ that has finite outer measure and for which $$m^*(\mathcal{O}\sim E) > m^*(\mathcal{O})-m^*(E).$$
Proof:
Since $E$ is not mensurable, then this implies that $\exists \epsilon > 0, \forall \mathcal{O} \supset E$ and $m^*(\mathcal{O}\sim E ) \geq \epsilon$. In particular, since $m^*(E) < \infty$, there must be some $\mathcal{O}$ with finite outer measure so that $\forall \epsilon>0,$ we get: $$m^*(E) + \epsilon > m^*(\mathcal{O})\Rightarrow m^*(\mathcal{O}) - m^*(E) < \epsilon$$ but, $$m^*(\mathcal{O}\sim E ) \geq \epsilon \Rightarrow m^*(\mathcal{O}) - m^*(E) \leq m^*(\mathcal{O}\sim E ),$$ as required.
In fact a stronger result holds:
Proof: Note that, since $m^∗(A)<\infty$ and $E\subseteq A$, we have $m^∗(E)<\infty$. So $E$ has a mensurable cover $D$ (it means $E\subseteq D$, $D$ is measurable and $m^*(D)=m^*(E)$).
Let $B=D\cap A$. It is easy to see that $B \subseteq A$ and that $B$ is also a mensurable cover of $E$. In fact, $E\subseteq B$, $B$ is measurable, and $m^*(E)\leq m^*(B)\leq m^*(D)=m^*(E)$, so $m^*(B)=m^*(E)$.
Since $A \sim E =(A \sim B) \cup (B \sim E)$, using the the sub-additivity of $m^*$, we have
$$ m^*(A \sim E)\leq m^*(A \sim B) + m^*(B \sim E)$$ So, since $m^*(A \sim B)\leq m^*(A) < \infty$ we have $$ m^*(B \sim E) \leq m^*(A \sim E)- m^*(A \sim B) $$ By hypothesis, we have that $m^∗(A\sim E) \leq m^∗(A)−m^∗(E)$, and since $m^*(B)=m^*(E)$, we have $$ m^*(B \sim E) \leq m^*(A \sim E)- m*(A \sim B) \leq m^∗(A)−m^∗(B) - m^*(A \sim B) $$ But $A$ and $B$ are measurable, and so is $A\sim B$, then we have (using again that $m^*(A) < \infty$): $$ m^*(B \sim E) \leq m^∗(A)−m^∗(B) - m^*(A \sim B)= m^∗(A)−m^∗(A)=0$$
So $m^*(B \sim E)=0$, which implies that $B\sim E$ is measurable. Since $E=B \sim (B\sim E)$, we have that $E$ is measurable.
Remark: You are trying to prove
If there is, at least, one open set $\mathcal{O}$ containing $E$ that has finite outer measure and is measurable, then the result you are trying to prove is an immediate consequence of the general result I proved above in this answer.
Remark 2 There are rather "pathological" examples where there is $E$ such that $m^*(E)<+\infty$ and $E$ is NOT measurable, but there is NO open set $\mathcal{O}$ containing $E$ such that $\mathcal{O}$ has finite outer measure and $\mathcal{O}$ is measurable. In such cases, the general result (proved above) remains true, but the result you are trying to prove may be FALSE.
Here is one such example: Let $X=\{0,1,2\}$, $\Sigma=\{\emptyset, \{0,1\}, \{2\}, X\}$ and $m$ be the measure defined by $m(\emptyset)=0$, $m(\{0,1\})=1$, $m(\{2\})=\infty$ and $m(X)=\infty$. It is easy to see that $\Sigma$ is a $\sigma$-algebra and that $m$ is a measure. Let $\tau= \{\emptyset, \{1\}, \{2\},\{1,2\},\{0,2\}, X\}$. It easy to see that $\tau$ is a topology on $X$.
We can prove that:
and that the $\mu^*$-measurable sets are precisely those in $\Sigma$.
Now consider $E=\{1\}$. We have that $m^*(E)=1$, and $E$ is NOT $\mu^*$-measurable.
Note that for all $\mu^*$-measurable set $A$ with $m^∗(A)<\infty$ and $E\subseteq A$, we have $$m^∗(A\sim E) > m^∗(A)−m^∗(E)$$ Proof: In fact, there is only one $\mu^*$-measurable set $A$ with $m^∗(A)<\infty$ and $E\subseteq A$, it is $\{0,1\}$, and it is to check that $$m^∗(\{0,1\}\sim E) = 1 > 0 = m^∗(\{0,1\})−m^∗(E)$$ So the general result remains true.
On the other hand, the only open set $\mathcal{O}$ containing $E$ such that $\mathcal{O}$ has finite outer measure is $E$ itself, that is to say, $\mathcal{O} =E$. But, for such $\mathcal{O}$, we have
$$m^∗(\mathcal{O}\sim E)=\mu^*(\emptyset)=0 \ngtr m^∗(\mathcal{O})−m^∗(E)=1-1=0$$ So the result you are try to prove is not true in this case.
(Note that there is NO open set $\mathcal{O}$ containing $E$ such that $\mathcal{O}$ has finite outer measure and $\mathcal{O}$ is measurable, so, in this case, the result you are try to prove does not follow from the general result.)
Summarizing: The result you are trying to prove is NOT true in general. To ensure it is true, we must assume (add a condition) that there is, at least, one open set $\mathcal{O}$ containing $E$ such that $\mathcal{O}$ has finite outer measure and $\mathcal{O}$ is measurable.
There several other conditions (for instance, the measure being outer regular) which would automatically imply that there is, at least, one open set $\mathcal{O}$ containing $E$ such that $\mathcal{O}$ has finite outer measure and $\mathcal{O}$ is measurable.