I would like to know if it is possible to give conditions under which $g(\theta)$ is twice-differentiable, where:
$$ g(\theta) = argmin_{\eta \in \mathcal{B}} \sum^n_{i=1}\left(y_i - \sum^R f(\theta,x_i,\alpha_r)\eta_r\right)^2 $$ and $$ \mathcal{B} = \left\{\eta \in \mathbb{R}^R: \sum^R \eta_r = 1 \;\;\text{and}\;\; \eta_r \geq 0 \right\} $$ I know that there exist related questions but none of them addresses this kind of problem with constraints. Moreover, I can tell that the objective function is well-behaved and the set $\mathcal{B}$ is also "nice" so I was wondering if there is a way of exploiting the properties of the objective function in the above program to find conditions that guarantee that $g(\theta)$ is twice differentiable (for example something that can allow me to interchange the derivative and the argmin). The Implicit Function Theorem might be a way to go but this will assure that $g(\theta)$ is only differentiable and I need more.
I should add that the properties of f(.), which I do not state for simplicity, guarantees that the problem has a unique solution (I have edited the problem with respect to my original post).
If we define $\zeta(\theta) = (f(\theta, x_1), \dots, f(\theta, x_r))$, then we can reinterpret the objective function as $\left(y - \zeta(\theta) \cdot \eta\right)^2$. Setting it equal to $0$ gives the equation (in $\eta$) of a hyperplane $\cal P$ of dimension $R - 1$ normal to $\zeta(\theta)$ and at a distance of $\frac{y}{\|\zeta(\theta)\|}$ from the origin. Further, the objective function itself is the square of the distance from the point $\eta$ to $\cal P$: $d^2(\eta, \mathcal P)$.
$\cal B$ is a simplex of dimension $R-1$ whose vertices are the various points on the positive axes a distance of $1$ from the origin. If $\cal P$ intersects $\cal B$, any point $\eta$ in the intersection will have $d^2(\eta, \mathcal P) = 0$. So by the condition that the objective function always has a unique solution, there cannot be more than one point in the intersection. By the linearity of $\cal P$ and convexity of $\cal B$, that point has to be a vertex. When they do not intersect, the unique point minimizing the distance to $\cal P$ also has to be a vertex. (By convexity, it has to be on the boundary, and by linearity and uniqueness it has to be a vertex.)
So if $g(\theta)$ is to be well-defined, it will be a vertex. Further, if $f$ is continuous in $\theta$, that vertex cannot change. If $f$ is continuous, so will be $\zeta$, as will be $d(v_1,\mathcal P) - d(v_2,\mathcal P)$, where $v_1$ and $v_2$ are two vertices of $\cal B$. If $v_1$ switches from being further than $v_2$ from $\cal P$ to being closer, somewhere in the middle the difference in their distances must cross $0$, and at that point the objective function is the same for both, meaning that $g$ would be undefined, contrary to the requirement on $f$.
Thus it only requires having $f$ continuous with respect to $\theta$ to ensure that $g$ is not only twice-differentiable, but actually constant. And since $g$ can only take on a finite number of values, being constant is the only way it can be continuous, much less differentiable.