I'm working on the following optimization: $$\min_{x,y,Z} f(x,y,Z) \\s.t.~~ g(x,y,Z)>0 \\ h(x,y,Z)=0$$
$\{x,y\}$ are vectors and $Z$ is a matrix and $f(x,y,Z)$ is convex and differential, but it is not separable in terms of $\{x,y,Z\}$.
I've decided to use the alternating optimization scheme or block coordinate descent, meaning to consider each of the variables at a time and repeat the loop until reaching the desired convergence point.
But i'm not sure whether to try gradient based approaches in which at each step of the big loop (for each variable) they try to optimize the variable only one step-size. Or is it fine if we try to solve the minimization problem for each variable each time?
For example at each step of the loop, i can solve the constrained optimization problem for each variable using a general non-linear solver while fixing the others. but i think this would prevent the problem to be converged to a point and instead just jumping around.
Could you please give me some insight into situations like the above?