How to model an if statement as a linear transformation

23 Views Asked by At

While working with the rectified linear activation function or ReLU, which can be mathematically expressed as ReLU(X) = max(0,X) where X

$$ X = \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{pmatrix}, \qquad x_i \in \mathbb{R} $$

I started thinking about the idea of a decision and how it is modeled in standard mathematics. Although I am familiar with the a notation such as:

\begin{cases} \frac{n}{2}, & \text{if $n$ is even} \\[2ex] 3n+1, & \text{if $n$ is odd} \end{cases}

I saw that in certain particular cases such a condition can be written as a single equation. For example, in the case of Support Vector Machines the conditions:

$$ f(\mathbf{x}_i) = \begin{cases} w^T\mathbf{x}_i + b \geq 1, & \text{if } y_i = +1 \\ w^T\mathbf{x}_i + b \leq -1, & \text{if } y_i = -1 \end{cases} $$

can be written as a single condition in the following way:

$$y_i(w^T\mathbf{x}_i + b) \geq 0$$

I was wondering if the ReLU function can be modeled using the same logic. For example I would want to find a matrix A, such that

$$ AX = \begin{pmatrix} x_1' \\ x_2' \\ \vdots \\ x_n' \end{pmatrix}, \qquad x_i' \in \mathbb{R} $$ where:

$$ f(\mathbf{x}_i) = \begin{cases} 0, & \text{if } x_i \leq 0 \\ x_i & \text{if } x_i > 0 \end{cases} $$

Does such a matrix exist or is this the whole idea of "non-linearities" in the context of activation functions and such a condition can be implemented only programmatically?