$\mathbf{X}^T \mathbf{X}\mathbf{A} - \mathbf{X}^T \mathbf{L} + \mathbf{\Gamma}\mathbf{A} = 0$
$X$ is $n\times m$
$\Gamma$ is $m\times m$ diagonal
$A$ is $m\times 1$
$L$ is $n\times 1$
In the problem, $m>>n$, Need to solve for $A$ and rest all are given. how can I solve this equation in the fastest possible way? I cannot afford to store an $m\times m$ matrix in the memory.
Note : $\Gamma$ is a diagonal matrix with positive entries.
Your solution will be
$$(\Gamma + X^T X)^{-1} X^T L$$
By the Woodbury identity $$ (\Gamma + X^T X)^{-1} = \Gamma^{-1} - \Gamma^{-1} X^T (I + X \Gamma^{-1} X^T)^{-1} X \Gamma^{-1} $$ so you want $$ \Gamma^{-1} X^T L- \Gamma^{-1} X^T (I + X \Gamma^{-1} X^T)^{-1} X \Gamma^{-1} X^T L$$
All this can be done step-by-step using only objects of at most $mn$ entries.