Suppose I have a function $f(x,y)$ where $f$ is convex in $x$ and $y$ but not in $(x,y)$. Suppose that for a fixed $y$, the minimization over $x$ lands on a solution that doesn't depend on $y$, e.g.
$$ \bar x = \arg\min_x\; f(x,y), \quad \forall y. $$
Then I also find $$ \bar y(x) = \arg\min_y \; f(x,y). $$ Is $(\bar x, \bar y(\bar x)) = \arg\min_{x,y}\; f(x,y)$?
Application: I'm trying to understand the maximization step in the Gaussian mixture models. Here, we want to find
$$ (\mu, C) = \arg\min_{\mu, C} \; \underbrace{\sum_i -z_i \log \det(C) + z_i (x_i-\mu)^TC (x_i-\mu)}_{f(\mu,C)} $$ The standard answer gives
$$ \bar \mu = \frac{\sum_i z_i x_i}{\sum_i z_i}, \qquad \bar C = \frac{\sum_i z_i (x_i-\bar \mu)(x_i-\bar \mu)^T}{\sum_i z_i} $$ which is exactly what would happen using the procedure I outlined above, but it doesn't seem like this function is jointly convex in $\mu$ and $C$. In fact, in the 1-D 1-sample case, the Hessian is $$ \frac{\nabla^2 f(\mu,C)}{z_i} = \left[\begin{matrix} C & x-\mu \\ x-\mu & C^{-2}\end{matrix}\right] $$ which, well, isn't necessarily positive semidefinite.
Maybe my interpretation is off?
Thanks all!