Remove unlogical points (noise) in a curve

Question

I have a time serie with 1000 values, but some values are out of logic. Let's say I have points like that:

[[...]100,101,100,99,97,103, 10,5,2, 100,101,102,97[...]]

(there are much more value in the time series) But how to remove this "noise" (10,5,2) ? Is there any mathematical way to do that ?

Thanks

Answer 1

You can use a low pass filter to cancel high frequency terms. Low-pass filters provide a smoother form of a signal, removing the short-term fluctuations, and leaving the longer-term trend. For the theory and reference, check wiki link

It helped me to cancel some noises in my program via the code below.(You may need to modify in your program language)

---

$ y[0] := x[0]$

for $i$ from $1$ to $n$

 y[i] := α * x[i] + (1-α) * y[i-1]

return y

---

where $ x[i] $ is your input series,

$ y[i] $ is your output series ,

$α$ used for cut-off frequency , select a real number between $0$ and $1$

But please note that, this code can change your other terms little bit too because of filter cutt-off frequency or non-ideal filter characteristics . You must find appropriate cutt of frequency and maybe you can need to use higher degree filter to get ideal filter behaviour.

You can also check that to see how to design a digital filter