Issue in carrying SVD on a large size matrix

294 Views Asked by At

I need to construct a lower-dimensional matrix $M$ of size $70,000 \times 70,000$ from a matrix of size $10^5 \times 10^5$ using SVD, discarding smaller singular values and the corresponding singular vectors.

I keep getting "out of memory" error.

Is it mathematically correct to split the matrix in two halves of size $50,000 \times 100,000$ each and carry SVD on individual halves and eliminate $15,000 \times 30,000$ from each by eliminating smaller singular values?

1

There are 1 best solutions below

4
On

If I'm correct, you have a matrix $A\in \mathbb{R}^{100k\times 100k}$ and are trying to find a matrix $B\in \mathbb{R}^{70k \times 70k}$ with SVD. This is, as it's currently stated, impossible, as truncation of singular values will only reduce the rank of $B$, not the outer sizes.

You could, however, find $B\in \mathbb{R}^{100k \times 100k}$, with a rank as low as you want. To calculate an SVD is rather expensive (especially if interested in $U$ and $V$). One way to go around this is to calculate a non-optimal low-rank-optimization (by Adaptive cross approximation or random sampling or similar) and then orthogonalize the low-rank factors.