My question is about how one defines (as a function) a square-well type of interaction potential between two particles in 3D space. More precisely, as an analogy, one may have other forms of pair-wise interactions, such as $V(\mathbf r) = e^{-r/r_0}$ where $r$ is the distance between the two particles (point particles assumed), and here the potential $V$ is a quite simple exponential decay.
In contrast, how is the square well case generally defined? I've had two ideas but I don't know if either one is correct: (with $r$ the distance between the points and $a,\epsilon$ two given positive constants.)
a) $$ \begin{cases} a & 0 < r < \epsilon \\ 0 & otherwise \end{cases} $$
or b) $$ \begin{cases} a & 0 < r_{x,y,z} < \epsilon \\ 0 & otherwise \end{cases} $$
Do either of these definitions correctly represent a square-well type of interaction?
Conventionally, in the field of interaction potentials, the square well potential is taken to be spherically symmetric (the "square" refers to the shape of the potential curve vs $r$, not to the geometry in 3D). So your answer (a) is closest to the usual one, but not exactly right. Firstly, $a$ should be a negative constant, not a positive one, otherwise it would be a "hump" not a "well". Secondly, it is almost always understood that the square well has been added to an infinitely repulsive hard sphere potential which prevents atoms from overlapping each other. So, most people would understand a square well potential to be (adopting your notation) $$ V(r) = \begin{cases} \infty & r < \sigma \\ -a & \sigma \leq r < \epsilon \\ 0 & r \geq \epsilon \end{cases} $$ with three parameters: $\sigma$ and $\epsilon$, the hard core diameter and well diameter respectively, both positive and with $\sigma<\epsilon$; and $a>0$ (but with an explicit $-$ sign in the formula above), the well depth.
You can find out more about this model, and references to the literature, on the SklogWiki.