Given a $ n \times m $ matrix $ \mathbf{A}$ whose rank I know is $r \ll n,m$, i.e.,
$$ \mathbf{A} = \sum_{i=1}^r \mathbf{a}_i \mathbf{b}_i^T $$
I would like to reconstruct the unknown column and row vectors $\mathbf{a}_i$ and $\mathbf{b}_i$ for $i > r$.
If I use the SVD to compute the low-rank approximation of $ \mathbf{A} $ all but the first $r$ singular values will be zero, and the following will hold:
$$ \mathbf{A} = \sum_{i=1}^r \mathbf{u}_i s_i \mathbf{v}_i^T = \sum_{i=1}^r \mathbf{a}_i \mathbf{b}_i^T $$
where $\mathbf{u}_i$ and $\mathbf{v}_i$ are the left and right singular vectors respectively and $s_i$ are the corresponding singular values. Yet the $\mathbf{u}_i$ will not directly correspond to the input vectors $\mathbf{a}_i$.
Is there a relationship of the sort $ \mathbf{a}_i = f( \mathbf{u}_1, \ldots, \mathbf{u}_r)$ that would allow me to set up an optimization problem for the reconstruction of the unknown $\mathbf{a}_i$ and $\mathbf{b}_i$?
All tips on how to approach this problem or literature references would be appreciated!