Let's assume we have three collinear variables/factors, $X_1$, $X_2$, $X_3$. Would there be a method to orthogonalize these variables in a simultaneous way: in other word, orthogonalizing them in such a way that the order of the variables doesn't matter?
To clarifi this, let's consider the following orthogonalization method where the order of the orthogonalization process actually matters:
Let's choose factor $X_1$. Now, we will regress $X_2$ with respect to $X_1$:
$X_2 = \alpha_3 + \beta^{(2)} * X_1 + \epsilon_2$
In this case, $\epsilon_2$ is the "pure" $X_2$ value: we have striped out any $X_1$ effect from $X_2$.
Now, we will orthogonalize $X_3$ with respect to the first two factors, $X_1$ and the pure factor $\epsilon_2$:
$X_3 = \alpha_2 + \beta^{(3,1)} * X_1 + \beta^{(3,2)} * \epsilon_2 + \epsilon_3$
Now, we obtain three orthogonal factors: $X_1$, $\epsilon_2$, and $\epsilon_3$.
However, the problem with this hierarchical methodology is that the order in which your orthogonalize (in this case $X_1$ first, then $X_2$, etc.) matters. In this case, $X_1$ has the most effect, following by $\epsilon_2$, then finally $\epsilon_3$.
Is there any methodology that would allow all variables to be orthogonalized with respect to each other, but in a simultaneous way, rather than hierarchically?
Thanks in advance!