Is there a way to get an average that weights each item inversely based on distance from the mean?

Question

I'm not sure how to phrase this, or even if such a thing exists. Sorry!

I have a bunch of data points, which are mostly pretty tight. Each group should hone in on a specific point in space. Unfortunately, some of the recordings are polluted with movements (i.e. a bump of the table). These movements always end where they started, so the data points that follow are similar to those that precede the movement.

These movements affect a very small proportion of the data points, so I've just been ignoring them so far and have just used the mean of the co-ordinates to determine the location. What I'd like is a function that weights each data point according to it's similarity to the others, such that a data point during a movement would have a smaller weighting than a static point.

Is there a name for this? How is it done?

Answer 1

Say your data points are $(x_i)_{i=1\dots n}$. You could define

$$\mu = \frac{1}{n}\sum_{i=1}^n x_i$$

to be the unweighted mean, and then define weights

$$w_i = \frac{1}{|x_i - \mu| + \epsilon}$$

for some $\epsilon > 0$ and finally define a weighted mean by

$$\mu' = \frac{\sum_{i=1}^n w_i x_i}{\sum_{i=1}^n w_i}.$$

The $\epsilon$ appears in the denominator of the second equation so that you don't end up with infinite weights.

I should note that this is a very ad-hoc solution, though, and doesn't correspond to any statistical method I've heard of.

Answer 2

Alternately, if you know the expected accuracy of the measurements, you can just ignore those too far away. If your normal accuracy is $\pm 1$ mm, just delete any that are not within that distance of enough others. Your model is that this data is just garbage (caused by bumping, for example). If all the bumps move it the same direction and you want the unbumped location, it is better to ignore the data taken when it has moved. If you have a time stame on each reading, you might even think about deleting all data that differs too much from the neighboring time stamps.

Answer 3

One simple approach would be the following:

1. Compute the mean of all of the data points.

2. Compute the standard deviation from the mean.

3. Throw out all of the data points that are more than two standard deviations from the mean.

4. Compute the mean of the new data.

If you want something for sophisticated, there are several standard tests for identifying outliers in a given data set. These include:

1. Chauvenet's criterion, which adjust the "cutoff" based on the number of data points. According to this criterion, a $2\sigma$ cutoff is appropriate for 10-point data set, a 3$\sigma$ cutoff is appopriate for a 200-point data set, and a $4\sigma$ cutoff is appropriate for a 10,000-point data set.

2. Grubbs' test, which iteratively removes outliers one at a time.

3. Dixon's Q Test, which identifies outliers based on the "gap" between an outlier and the closest other data points.

Answer 4

There are many ways to weigh points for purposes of correlation, clustering, regression etc.

Robust methods are those that tend to weigh outliers less (or even discard them altogether) than classical methods - although even robust methods date back to Gauss and Laplace.

Gauss favored a least-mean-squared approach in which we would call classical or $L_2$ metric, aka Euclidean distance. This norm weighs outliers (actually all points) quadratically as a function of distance from the mean.

Laplace developed a least-absolute-deviations approach associated with the $L_1$ metric. This metric is called robust because points (including outliers) are weighed linearly in relation to the mean.

Off the bat I don't know if any $L_p$ norm will yield an inverse weighing (But there are hundreds of families of named metrics, see eg Deza & Deza "Dictionary of Distances")

Related to the robust methods are the important concepts of median, order statistic, breakdown point and truncated mean.