How does one guarantee $ f(h) = max(0 , W^T h + b ) $ to be piece wise linear in a bounded positive hypercube?

62 Views Asked by At

Consider:

$$ f(h) = max(0, W^T h + b ) = max( 0, g(h,W,b) )$$

the goal is to choose $W$ and $b$ such that given a bounded (positive) domain the function $f(h)$ is piece wise linear in the given domain (note the region is fixed a priori and cannot be changed). I thought about this a little and it seems that it equivalently means that $f(h) = 0$ for some region or even better $g(h,W,b) < 0$. Ideally $h \in \mathbb{R}^D$ and $h$ is bounded, for example each coordinate $h_i \in [0, K]$ for some constant $K \in \mathbb{R}$.

To start this off I consider a case that I could visualize: $h \in \mathbb{R}^2$. I considered the bounded region $h \in \mathbb{R}^2$ Thus we have:

$$ f_2(h) = max(0, W_1 h_1 + W_2 h_2 + b ) = max( 0, g_2(h,W,b) ) $$

In this case its quite easy to do the task because one can even just go to mathematica a visualize the problem and choose the numbers so that there is a region that its linear and another that is not. I quickly noticed that given only a positive hypercube one could set $b = 0$ (or really small) and then just have $W_1, W_2$ opposite signs and not differ in magnitude to much (if they differ by a lot then $-W_2$ would eat up the values and drive them to be negative which would be driven to zero by the max, similarly if $W_1$ is to large then because the max never sees a negative value it ends up just being a linear function). Given a fixed bound its not to hard to play around with this simple case.

However, when it comes up to a more interesting case like $\mathbb{R}^4$ (the one I am mostly interested) I found it harder to reason how to choose $W^T$ and $b$. Is there a systematic to tackle this problem given a fixed positive hypercube to choose $W$ and $b$ such that there is always a region that is piece wise linear? If its not possible is it at least possible to guarantee that some region is piece wise linear or maximize the area/volume where the function is piece wise linear?

Obviously my worry is that if the region is enclosed in such a way that $f(h)$ is always positive so it doesn’t have a region where its zero (or similarly its always zero and never linear). Given a positive hypercube, is it possible to guarantee that $f(h)$ has some region that is zero? I'm mostly interested right now at 4D but anything that generalizes would be fantastic/excellent!

Essentially as the comments have pointed out, a different way to express the question is to require the kink of $f(x)$ to lie inside the prescribed hypercube.