How does LASSO select the heavy collinear features (randomly?)

Question

We know that for a group of heavy collinear features, LASSO will mostly to select one of them and set the others zero. And many references said that this selected feature is randomly determined by LASSO. I am not sure for this conclusion and don't know the reason. Could you offer some related references?

Answer 1

I think you are making a small confusion here.

For a fixed data vector $y$ and a matrix $A$, the number of solution to the LASSO problem, defined as

$$\min_x ||Ax-y||_2+\lambda ||x||_1$$

is either one or infinite. In both cases, the strongest feature (eventually plural) is never chosen at random. The process is entirely deterministic.

However, when computing the solution numerically, depending on the algorithm, the initialization and the tolerance required on the solution. The results might be different.

Furthermore, if the data vector is the result of an experiment, then randomness is involved in the determination of $y$. Assuming $A$ contains strongly colinear features, the result might be unstable as well so several outputs of the same experiment might yield different results.

As a side note, if it happens, the selection among the colinear features is unlikely to be uniform. Strictly speaking, being randomly selected does not necessarily means that the different outomes have equal probability to happen.