What if coordinate descent updates multiple variables?

Question

The coordinate descent method update variable one at a time. For instance, if the variable $x_i$ is selected to be updated based on Gauss-Southwell rule, then $x_i^{(k)}=\text{argmin}_{x_i}\ f(x_1^{(k-1)},\dots,x_i,\dots,x_n^{(k-1)})$.

Is there an algorithm that updates multiple variables in each iteration? For instance, if $x_i$ and $x_j$ are chosen to be updated, then $x_i^{(k)},x_j^{(k)}=\text{argmin}_{x_i,x_j}\ f(x_1^{(k-1)},\dots,x_i,x_j,\dots,x_n^{(k-1)})$.

*$k$ denotes the iteration

Answer 1

Yes. It is called Block Coordinate Descent.

You may read about it at:

• Analysis of the Block Coordinate Descent Method for Linear Ill Posed Problems.
• Convergence of Block Coordinate Descent with Diminishing Radius for Non Convex Optimization.
• Stanford - CME307/MS&E311: Optimization - Block Coordinate and Stochastic Gradient Decent Methods.

Basically it is like Jacobi iteration vs. Gauss Seidel iteration.