Using this paper as a reference (Section IV.C, page 4318), We have the following objective function which we wish to minimize with respect to $D \in \mathbb R^{n \times K}$:
$$\min_{D} \{||Y-DX||_F^2\} \quad \text{subject to} \quad ||X||_0 \leq T_0$$
Which could be simplified as follow:
$$||Y-DX||_F^2\\=||Y-\sum_{j=1}^K d_jx_T^j||_F^2\\=||\big(Y-\sum_{j \neq k} d_jx_T^j\big)-d_kx_T^k||_F^2\\=||E_k-d_kx_k^T||_F^2$$
Then assume that both $D$ and $X$ are fixed and only one column of D and its corresponding row in X remain in question. The authors then suggest to apply SVD on $E_K$ to get an approximation of $d_k$ and its corresponding row in $X$ and this is where I have problem understanding. Why do we choose SVD and why do we define the desired column of $D$ as the first column of $U$ in $$E_k=U \Delta V^T$$ and the coefficient vector as the first column of $V$ multiplied by $\Delta (1,1)$? A clarification would be highly appreciated.
The authors use the SVD for the following reason. Let $A$ be any matrix. Then, for any matrix $B$ of rank at most $k \leq \mathrm{rank}(A)$ we have
$$ \|A - A_k\|_F \leq \|A - B\|_F, $$
where $A_k = \sum_{i=1}^k \sigma_i u_i v_i^T$ (sum of the first $k$ terms of the SVD expansion of $A$). That is, $A_k$ is the best rank $k$ approximation of $A$. This result is referred to in the literature as the Eckart-Young theorem.
Now consider the problem of choosing $d_k$ and $x_k$ to minimize $\|E_k - d_k x_k^T\|_F$. By the Eckart-Young theorem
$$ \|E_k - \Delta(1,1) u_1v_1^T\|_F \leq \|E_k - B\|_F $$
for all rank one matrices $B$. Since $d_kx_k^T$ is at most rank one it follows that we should set $d_kx_k^T = \Delta(1,1) u_1v_1^T = u_1(\Delta(1,1) v_1)^T$. Therefore, let $d_k = u_1$ and $x_k = \Delta(1,1)v_1$.