Why can an element of a Hilbert space be written as this sum?

Question

From Understanding Machine Learning:

Let $w^*$ be an optimal solution of this equation:

$$\min_w (f(\langle w, \psi(x_1)\rangle, \dots , \langle w, \psi(x_m) \rangle) + R(||w||))$$

where $\psi:X \rightarrow \text{Hilbert Space (H)}, x_i \in X,$ and $ R : \Bbb R_{+} \rightarrow \Bbb R$ is a monotonically nondecreasing function.

Then $w^*$ can be written as $w^* = \sum_{i=1}^m\alpha_i\psi(x_i)+u$ where $\langle u,\psi(x_i) = 0 \rangle$ for all $i$.

Why can $w^*$ be written in this way?

Answer 1

You can always write $w=w_a+w_b$ where $w_a$ is in the span of $\lbrace \psi(x_i)\rbrace_{i\in[m]}$ and $w_b$ is orthogonal to this space.

Then you get that $\langle w,\psi(x_i)\rangle = \langle w_a,\psi(x_i)\rangle$ and $\lVert w \rVert=\sqrt{\lVert w_a\rVert^2 + \lVert w_b\rVert^2}\geq \lVert w_a\rVert$ with equality if and only if $w_b=0$. Hence, since $R$ is monotically nondecreasing, the minimum can only be achieved if $w_b=0$ which proves your statement.