This is Exercise 47(b) from Chapter 1 of Lang's Algebra. In part $(a)$ one is asked to show that $G$ is doubly transitive if and only if $H$ is transitive on $S\setminus\{s\}$. I've been able to prove that part. As for the title in the question, I've been able to prove a partial result. Namely, $$G=H\cup HgH$$ where $g\notin H$.
I've looked a few references but I can find the result of the exercise. Decomposing $G$ into the double cosets is the best result I can find.
Let $t$ be any element of $G \setminus H$. Then, since you have already proved that $G = H \cup HgH$, we must have $t \in HgH$ and hence $HgH=HtH$. Let $T = \langle t \rangle$. Then $1 \in T$ and so $H = H1H \le HTH$, and also $HgH = HtH \le HTH$, so $G=HTH$.
This applies in particular when $t$ and $T$ have order $2$. The fact that $G$ is doubly transitive implies that we can conjugate an involution in $G$ to an involution that is not contained in $H$, so such elements $t$ of order $2$ exist.