I need to construct a lower-dimensional matrix $M$ of size $70,000 \times 70,000$ from a matrix of size $10^5 \times 10^5$ using SVD, discarding smaller singular values and the corresponding singular vectors.
I keep getting "out of memory" error.
Is it mathematically correct to split the matrix in two halves of size $50,000 \times 100,000$ each and carry SVD on individual halves and eliminate $15,000 \times 30,000$ from each by eliminating smaller singular values?
If I'm correct, you have a matrix $A\in \mathbb{R}^{100k\times 100k}$ and are trying to find a matrix $B\in \mathbb{R}^{70k \times 70k}$ with SVD. This is, as it's currently stated, impossible, as truncation of singular values will only reduce the rank of $B$, not the outer sizes.
You could, however, find $B\in \mathbb{R}^{100k \times 100k}$, with a rank as low as you want. To calculate an SVD is rather expensive (especially if interested in $U$ and $V$). One way to go around this is to calculate a non-optimal low-rank-optimization (by Adaptive cross approximation or random sampling or similar) and then orthogonalize the low-rank factors.