Convex optimization - bound on a subgradient

Question

Consider the following optimization problem: $$ \min_{b\in \mathbb{R}^d} \sum_{i=1}^n |y_i-x_i^{\top}b|. $$ A text I'm reading states without a proof that $$ \left\|\sum_{i=1}^n \operatorname{sgn}(y_i-x_i^{\top}b^*)x_i\right\|\le \left\|\sum_{i=1}^n 1\{y_i=x_i^{\top}b^*\}x_i\right\|, $$ where $b^*$ minimizes the sum of absolute deviations. It there a way to prove the inequality? (I know that when the points ($y_i,x_i$) are distinct, $\sum_{i=1}^n 1\{y_i=x_i^{\top}b^*\}=d$.)