First of all, I'd like to say that this is the intro to a homework problem. Please do not post any answers, I am only looking for clarification on some terminology in the setup.
I am trying to understand how this algorithm works and most of it makes sense except for the part where they "update the residual by regressing out the currently selected features". What does that mean? and what is the least squares regression of $y$ onto $X_{S_t}$?