A max-affine function is defined as the maximum over a set of affine functions, which is always convex. More specifically, we define a $K$-max-affine function $f:\mathbb{R}^d\to\mathbb{R}$ that can be written as $$ f(x) = \max\{\alpha_1^T x+\beta_1,\ldots,\alpha_K^T x+\beta_K\} $$ with $\alpha_i\in\mathbb{R}^d$ and $\beta_i\in\mathbb{R}$ for all $i = 1,\ldots,K$. Without loss of generality, we consider in this case the function $f$ defined on a compact space $[0,1]^d$, and we define a $B$-bounded and $L$-Lipschitz max-affine function with $K$ pieces to be $f$ such that $\sup_x\|\partial f(x)\| \leq L$ and $\sup_x |f(x)| \leq B$ for all $x\in[0,1]^d$, where $\partial f(x)$ denotes the subdifferential of $f$ at $x$.
It is known that any $B$-bounded and $L$-Lipschitz max-affine function with $K$ pieces can be represented by a single $K$-layer ReLU network with positive weights. This is due to the fact that the composition of convex and increasing functions is also convex. However, the set of functions that is reachable by such a neural network with a bounded parameter space often form a superset set of $B$-bounded and $L$-Lipschitz max-affine functions with $K$ pieces. For example, even a two-layer fully-connected network with width $K$ can represent a piece-wise linear function with $2^K$ pieces, which could be neither $L$-Lipschitz nor $K$-max-affine.
I am wondering whether we could construct a neural network (e.g., a two-layer nn), such that, when paired with a proper parameter space, it represents all the $B$-bounded and $L$-Lipschitz max-affine functions with $K$ pieces and nothing more.