Solving the equation with sum of max functions (ReLU)

Question

Given real constants $y$, $w_1$...$w_n$, $b_1$...$b_n$, is it possible to solve the unknown $x$ in closed form in the below equation? $$ y = \sum_i \max\{w_ix + b_i, 0\} $$

Answer 1

You can start using that:

$$ \max\{x,y\}=\frac{x+y+\vert x-y \vert}{2}.$$

In particular, you have that:

$$ \max\{x,0\}=\frac{x+\vert{x}\vert}{2} $$

Then $$\displaystyle{y=\sum_{i=1}^{n}\frac{w_{i}x+b_{i}+\vert{w_{i}x+b_{i}}\vert}{2}}$$

You need to observe that for each $i$ you have two cases:

$$\frac{w_{i}x+b_{i}+\vert{w_{i}x+b_{i}}\vert}{2}=\begin{cases} w_{i}x+b_{i} & \text{if}\ \ w_{i}x+b_{i}\geq 0, \\ \quad 0 & \text{if}\ \ w_{i}x+b_{i}< 0.\end{cases}$$

Because:

$$ w_{i}x+b_{i}<0 \Rightarrow \vert{w_{i}x+b_{i}}\vert=-(w_{i}x+b_{i}) \Rightarrow \frac{w_{i}x+b_{i}+\vert{w_{i}x+b_{i}}\vert}{2}=0 $$ and $$ w_{i}x+b_{i}\geq 0 \Rightarrow \vert{w_{i}x+b_{i}}\vert=w_{i}x+b_{i} \Rightarrow \frac{w_{i}x+b_{i}+\vert{w_{i}x+b_{i}}\vert}{2}=w_{i}x+b_{i} $$

Then your sum reduces to:

$$y=\sum_{i\,\vert\ w_{i}x+b_{i}\geq 0}\left(w_{i}x+b_{i}\right).$$

since $x$ does not depend on $i$ you can write the last equation like that:

$$ y =\left(\sum_{i\,\vert\ w_{i}x+b_{i}\geq 0}w_{i}\right)x+\left(\sum_{i\,\vert\ w_{i}x+b_{i}\geq 0}b_{i}\right)$$

From there you can isolate $x$ at least when the sum $\displaystyle{\sum_{i\,\vert\ w_{i}x+b_{i}\geq 0}w_{i}}$ is not zero. In fact:

$$ x = \frac{\displaystyle{y-\sum_{\{i\,\vert\ w_{i}x+b_{i}\geq 0\}}b_{i}}}{\displaystyle{\sum_{\{i\,\vert\ w_{i}x+b_{i}\geq 0\}}w_{i}}}.$$

Answer 2

Or you can simply say:

$$ y = \sum_{i=1}^{n}\max\{w_ix + b_i, 0\} = \sum_{i\,\vert\, w_{i}x+b_{i}\geq 0}(w_{i}x + b_{i})=\left(\sum_{i\,\vert\, w_{i}x+b_{i}\geq 0}w_{i}\right)x+\left(\sum_{i\,\vert\, w_{i}x+b_{i}\geq 0}b_{i}\right). $$

because, for each $k$ such that $\ w_{k}x+b_{k}<0\ $ the value of $\ \max\{w_{k}x + b_{k}, 0\}\ $ vanish. That's why we have: $$ \sum_{i=1}^{n}\max\{w_ix + b_i, 0\} = \sum_{i\,\vert\, w_{i}x+b_{i}\geq 0}(w_{i}x + b_{i}). $$