Assume you want to solve an optimization problem:
$$ w^* = \arg \min J(w)$$
where $w \in \mathbb{R}^M$ and $J(w)$ is non-linear and non-convex.
In a particular scenario that I am considering there are two possibilities for solving the above optimisation problem:
1) Just directly solve the above problem
2) Cast the problem as two nested optimisation problems. The optimisation variables $w$ are split into $w_1$ and $w_2$ such that $w = \lbrace w_1 , w_2 \rbrace$. Then solve:
$$ w_1^* = \arg \min J(w_1 | w_2^*) $$
where
$$ w_2^* = \arg \min J_2(w_2 | w_1)$$
I hope I haven't messed up the notation :). What I am trying to describe is that instead of optimising directly for the entire $w$ we only optmize a subset of it, $w_1$. The optimal value of the rest of the parameters, $w_2$, is computed "internally", i.e. for each set of $w_1$ values. One may argue that optimising $w_2$ that way leads to solving a different optimisation problem overall, so let's assume that $J_2$ is approximately equivalent to $J$.
The reasoning behind considering approaches like (2) is that essentially instead of having to search over the entire space of $w$ I hope that I am approximately solving the same problem by searching smaller subspaces sequentially as I described.
the question:
I wonder whether there is relevant literature on such nested optimisation approaches and relevant keywords so that I can further investigate.