I an intricate with something.
Let $S$ be a $C^2$-hypersurface that is the boundary of a convex and compact set $C$ in $\mathbb{R}^n$ with $n >1$. Let $0 \in int(C)$ and define a function $F: \mathbb{R}^n \to \mathbb{R}$ that is $\alpha$-homogeneous, with $\alpha >1$ and $F(x) = 1$ if, and only if, $x \in S$.
My questions: $F$ is strictly convex?
And can I say That, $F$ is twice differentiable at $\mathbb{R}^n-\{0\}$? And $F''(x) \geq 0$, $x \in \mathbb{R}^n-\{0\}$? Where $''$ indicates derivative in $x$.
I searched on the internet and some literature a little and could not find anything that indicates that this is true.
Thank you for your help.
I'm going to assume that the convex set $\text{conv}(S)$ you start with is bounded and contains $0$, because otherwise the definition of $F$ is a bit funky. But under these assumptions, for every $\mathbf x \in \mathbb R^n - \{0\}$, there is a unique $\mathbf s \in S$ such that $\mathbf x = \lambda \mathbf s$, where $\lambda > 0$.
Asking for your function $F$ to be $\alpha$-homogeneous means defining $F(\mathbf x) = \lambda^\alpha$, where $\lambda$ is the value found above. I'm going to show that more is true: if $h$ is any increasing convex function on the positive reals, then defining $F(\mathbf x) = h(\lambda)$ will make $F$ convex.
We'll need the following lemma from affine geometry: in the diagram below, $$\frac{OB'}{OB} = \frac{B'C'}{A'C'}\cdot \frac{OA'}{OA} + \frac{A'B'}{A'C'}\cdot \frac{OC'}{OC}.$$
I'm not going to prove the lemma, but it's not hard to do with standard geometry tools (or with coordinates).
Let $\mathbf x, \mathbf y \in \mathbb R^n$ be two points, and let $0 < t < 1$. We want to show that $$F(t \mathbf x + (1-t)\mathbf y) \le t F(\mathbf x) + (1-t) F(\mathbf y).$$
To this end, pick $A', B', C'$ in the diagram above to be $\mathbf x$, $t \mathbf x + (1-t)\mathbf y$, and $\mathbf y$; let $O$ be the origin. Let $A$ and $B$ be the points of $S$ on lines $OA', OC'$. By the convexity of $\text{conv}(S)$, $B \in \text{conv}(S)$.
By convexity of $h$ and the lemma, $$h(\frac{OB'}{OB}) \le \frac{B'C'}{A'C'}h(\frac{OA'}{OA}) + \frac{A'B'}{A'C'} h(\frac{OC'}{OC}).$$ Here, $\frac{B'C'}{A'C'} = t$, $h(\frac{OA'}{OA}) = F(\mathbf x)$, $\frac{A'B'}{A'C'} = 1-t$, and $h(\frac{OC'}{OC})$, so the RHS of this is $t F(\mathbf x) + (1-t) F(\mathbf y)$.
On the other hand, by convexity of the set $\text{conv}(S)$, $OB$ intersects $S$ somewhere at a point $B^*$ with $OB^* > OB$. Therefore $F(t \mathbf x + (1-t)\mathbf y) = h(\frac{OB'}{OB^*}) \le h(\frac{OB'}{OB})$, where we use the fact that $h$ is increasing. This shows that $F$ is convex.
In your question, you ask for strict convexity, but the condition you mean by that is doubly unclear. First of all, $F''(\mathbf x)$ is a weird thing to write. Do you mean the Hessian matrix of $F$? Are you asking for this to be positive semidefinite? Second, this second derivative check only corresponds to convexity, not strict convexity.
In fact, if $S$ contains line segments, it's easy to see that by the usual definition of strict convexity, $F$ will not be strictly convex (because $F$ will be $1$ along line segments in $S$). But if $S$ does not contain line segments, and if $h$ is either strictly convex or strictly increasing, we do get strict convexity.