I'm trying to figure out a way to construct a formula or method for the following process:
If Point E is inside the smaller rectangle (red area), the value should be 1.
If Point E is outside the bigger rectangle (blue area), the value should be 0.
If Point E is inside the bigger rectangle, but outside the smaller rectangle (white area), the value should be linearly interpolated between 1 and 0, based on the "difference" of the two rectangles.
The purpose of this is to fade out the edges of a rectangle shaped spotlight in OpenGL.
Any help would be appreciated.
Since according to your comment the rectangles are described by half their width and height, I assume that they are centered at the origin and aligned with the axes. So the smaller rectangle is $-w_1\le x\le w_1, -h_1\le y\le h_1$ and the larger likewise for $w_2,h_2$. Assume for the moment that $y=0$. Then the smaller rectangle is characterized by $\lvert x\rvert=w_1$ and the larger by $\lvert x\rvert=w_2$. In the latter case you want a result of zero, so start with $f(x,y)=w_2-\lvert x\rvert$. To achieve a value of $1$ at the inner rectangle, change this to $f(x,y)=\frac{w_2-\lvert x\rvert}{w_2-w_1}$. Now you have the linear gradient between these two values, but you still need to clamp things down so you don't exceed the $[0,1]$ range. Use $f(x,y)=\min\left(1,\max\left(0,\frac{w_2-\lvert x\rvert}{w_2-w_1}\right)\right)$ to achieve this clamping. And what about the $y$ coordinate? You could obtain an expression of the same kind for that. Then you can either multiply both expressions, or take the minimum. I'm not sure whether you'd call the former linear, since it's linear in each $x$ and $y$ but since it contains a $xy$ term I'd rather call it quadratic. So I'd go for
$$f(x,y)=\min\left(1,\max\left(0,\frac{w_2-\lvert x\rvert}{w_2-w_1}\right),\max\left(0,\frac{h_2-\lvert y\rvert}{h_2-h_1}\right)\right)$$
Notice that OpenGL has a
clampfunction, which will take care of that aspect, and aminfunction to combine the values.