I have a master equation for $(x,y,z)$ with the constraint $x+y+z=N$. $x$ can be regarded as the number of animal of a certain species in the whole system. In other words, I have a differential equation of the form $$ \dfrac{dP(x,y)}{dt}=\cdots, $$ where the RHS is too complicated to write down here.
Because there are only two independent variables, we can have ternary plot of $P(x,y)$ like (Please ignore the y-axis. it is wrong):
Now please take a look at the following ternary plots for $P(x,y)$ at different times:
It looks like that the probability mass is broken into three pieces and the three pieces are then attracted by three "attractors" (one at the corner, the other two are mid-points of the two edges). Of course, while the probability mass is travelling towards the attractors, the "shape" of the probability mass also change, which makes the problem difficult. The three sections (separated by the white lines) roughly are the basins of the three attractors. As I change the parameters of the system, the boundaries of the three sections will also change.
Note that the white lines are from population of infinite size, and I want to see how the finite population/system deviates from its infinite counterpart. For example, an initial probability distribution near the boundaries of two or three basins of attraction will be broken up into several pieces. Therefore, points near the boundary of the basin (of infinite system) in finite system are not really in the basin.
Once I know the locations of the attractors (the attractors are not individual points since this is a master equation), I want to know how much a point in the triangle plane deviates from being the basin of attraction in the corresponding infinite size system.
My question (sorry it is a little vague at the moment): What is the quantity $u(x,y)$ I should use to show how much $(x,y)$ deviates from being inside of the basin of attraction for a certain attractor? (for example, the attractor can be the one corner of the triangle)