I have a sensor whose trends I am attempting to analyze. The sensor samples values periodically from a system, but is subject to measurement disruptions of certain types. Visually, I know what the trend should look like, but I need to automate the analysis.
Below, I have a sample of some of the common noise patterns (the red trend indicates the sensor data, while the dashed grey trend indicates the true system state):
- First, we see spurious spikes. Both low and high spikes are possible.
- Next we see an intermittent shift where points are following the general slope but are offset by some constant
- Finally, we see a more sustained shift
We might model the sensor as a sum of two functions:
Sensor(t) = Real(t) + Noise(t)
If we do that, we can see that the output is a superposition of the following two trends:
I am looking for a reliable way to filter out this kind of stairstep noise. I can't use simple derivatives directly to test for the noise, because the sampling rate reduces the apparent sharpness of the derivative in many cases. Also, due to the sustained nature of some shifts, I can't use simple averaging. Exponential filtering is not sufficiently aggressive. To make matters worse, the real data is not actually as clean as it appears here, and may have its own real transients which sometimes approach the scale (in derivative) that is produced by the stairstep noise.
I have no control over the sensors, and I have no access to any additional data for checking. My filtering experience as an EE is mostly linear, and that doesn't apply well here as far as I can tell. Any suggestions would be appreciated.