score the evenness spread of data across time

Question

I'm trying to feature engineer a data frame with a time delta in seconds column. I would like to create a new feature that will score higher for data that spreads more evenly across time.

I've created this sample DF with 2 time lines, one is evenly distributed while the other one is not:

df = pd.DataFrame({'id':[1,1,1,1,1,2,2,2,2,2],'timestamp':[
            '2020-09-01 18:14:00',
            '2020-09-01 18:14:01',
            '2020-09-01 18:14:02',
            '2020-09-01 18:14:03',
            '2020-09-01 18:14:04',
            '2020-09-01 19:14:05',
            '2020-09-01 19:14:16',
            '2020-09-01 19:14:18',
            '2020-09-01 19:14:30',
            '2020-09-01 19:14:59'                                                
]})

df['timestamp'] = pd.to_datetime(df.timestamp)
df['delta'] = (df['timestamp']-df['timestamp'].shift()).fillna(pd.Timedelta(seconds=0)).astype('int64') / 1000000000

I've googled around and I'm kinda lost, is standard deviation the way to go?

Answer 1

Well, that depends on how severely you want to punish the "non-uniformity" and whether you want some extra features from your scoring method. I would use some extremal problem. For instance, if we have $n$ points on the circle of length $1$ with pairwise distances $d_1,\dots, d_n$, then the sum is fixed (to $1$) but the sum of the squares is at least $1/n$. So, if your time interval is of length $T$ and the successive distances from each point to the next one (wrapping the interval around into a circle, so you can go from the rightmost point to the leftmost one for the last distance) are $D_1,\dots D_n$, it would be natural to put $d_j=D_j/T$ and to define $V=n\sum_{j=1}^n d_j^2-1$ and put $\rm{Score}=e^{-\lambda V}$, say, so the score of $1$ is given to a perfect equi-spaced configuration and everything else would get it between $0$ and $1$.

The choice of $\lambda$ (which can also be made depending on $n$) will determine how sensitive you want your scoring function to be. I would put it at $1$ to start with and see whether you like the result it gives or not. Probably you will not like it but then you'll understand what exactly you are unhappy about and we can modify the definition of $V$ (which is technically any reasonable function of positions with unique global minimum at the equi-spaced configurations) to eliminate some particular effects.

In general, it is always this way for a problem with a vague objective: just start with something that is not too outlandish and modify it a few times to get its properties closer and closer to those of the "ideal scoring function" you have in mind. Of course, it would be faster if you could formulate all those ideal properties clearly in advance, but very few people can do it immediately and I doubt you are one of them, so just figure out what you really want by trial and error.