Let $(M,g)$ be a Riemannian manifold and $S\subset M $ open and strongly convex, i.e. any two points $p,q\in \bar S$ are connected by a unique minimizing geodesic with interior completely lying in $S$.
Conjecture: There exists an open neighbourhood $S'$ of $\bar S$ which is also strongly convex.
New question (March 20): It seems like the conjecture is false in general and user Dap has suggested a counterexample as an answer. Can you find another counterexample, maybe a simpler one?
Some remarks and proof attempts:
For sets which are merely convex this is not true. For example an open hemisphere of $S^2$ cannot be enlarged without losing convexity. The strongly convex sets of $S^2$ on the other hand have their closure strictly contained in a hemisphere and thus can obviously be enlarged to a new strongly convex set. This picture (respectively the ones that I can image on other surfaces) leads me to the assumption that the question has a positive answer.
Strongly convex sets are totally normal, (i.e. for every point $p\in S$ there is an open set $V\subset T_pM$ such that $\exp_p\colon V\rightarrow S$ is a diffeomorphism). My first idea was to first find an $S''\supset \bar S$ totally normal and then try to restict it to a strictly convex set. But just working with total normality cannot work: $S^2 \backslash p_0$ is totally normal, but cannot be extended to a larger totally normal set. Unfortunately I don't know how strong convexity can be incorporated from the start.
Edit (March 17): Here is a false counterexample, that was posted as an (now deleted) answer. I find it interesting neverletheless: Let $M=\{(x,y)\vert y>-e^x\}\subset \mathbb{R}^2$ and $S$ the open upper half plane (which is only convex and not strongly convex). If $S'$ is a open set containing $\bar S$, then $(0,-\epsilon)\in S'$ for some $\epsilon >0$. Take $x> 0$ so large that $\epsilon> xe^{-x+1}$, then the straight line between $(-x,0)$ and $(0,-\epsilon)$ contains the point $(-x+1,-\epsilon/x)$, which lies outside of $M$. In particular $S'$ cannot be convex.
I believe you can get an example from taking $M$ to be the elliptic paraboloid $z=x^2+y^2,$ with the Riemannian metric coming from $\mathbb R^3,$ and $S=\{p\in M\mid p_z<C\}$ for a particular constant $C.$
$C$ needs to be chosen so that there are conjugate points lying on the boundary of $S.$ There is a description of the conjugate points at "Example for conjugate points with only one connecting geodesic" and a description of the geodesics in do Carmo's "Differential Geometry of Curves and Surfaces", in terms of Clairaut's relation. The sets $\{p\in M\mid p_z\leq c\}$ are geodesically convex for any $c,$ with the minimizing geodesics going via $p_z<c,$ and the minimizers are unique for $c\leq C.$ As discussed in that MSE answer I linked to, this is an example where there are conjugate points not coming from a family of geodesics.
To show the strong convexity:
By Clairaut's relation, if we represent a non-meridian geodesic in co-ordinates $(r(t)\cos \theta(t),r(t)\sin\theta(t), r(t)^2),$ then $\theta(t)$ can be chosen to be an increasing function, and $dr/d\theta$ is some analytic function $f(r,\theta,c).$
By the identity theorem for analytic functions, there cannot be a continuous family of geodesics passing through two points.
A minimizing geodesic behaves sensibly in terms of longitudes. If a geodesic goes from polar co-ordinates $(r_1,\theta_1)$ to $(r_2,\theta_2)$ with $\theta_2>\theta_1+\pi,$ then by reflecting in the meridian going through longitudes $\theta_1,\theta_1+\pi,$ we can get a new curve that stay within the range of angle $[\theta_1,\theta_1+\pi].$ By smoothing this curve slightly we get a shorter curve. So the original geodesic could not have been minimizing. This means it makes sense to represent geodesics in polar co-ordinates.
Suppose for contradiction that there are distinct points $p,q$ in $\overline{S}$ joined by two minimizing geodesics. Pick $p_z<C$ if possible, and if the two points can be joined by a meridian then pick this as one of the geodesics - this allows a consistent choice of polar co-ordinates. Starting from $p,$ shoot a geodesic at an angle bisecting the two geodesics. This will hit one of the original two geodesics. This means we get a new pair of geodesics between some two points. The new endpoint must be $q,$ because otherwise we would get a pair of geodesics between two points contradicting the choice of $C,p,q.$ This kind of construction produces a family of geodesics between $p,q,$ contradicting the fact that there are no continuous families of geodesics joining $p,q.$