I have a sample of graphs (more than 10000...).
that look like in the image below:

I am searching for an unsupervised learning algorithms that can help me to detect anomalous observations.
Here what I suggest for beginning: for every observation I have a collection of points $(x,y)$. With this collection, I find Fourier series with regression (I compute coefficients with the base $\{1,\sin(x),\cos(x),\sin(2x),\cos(2x)\dots\}$). Now I have a set of coefficients instead of waves.
Somebody have an idea how to detect anomaly?
FFT (Fast Fourier Transform) each wave to get its Power spectral density (PSD).
Your "anomalous" waves may transform to a PSD like that of white noise, flat (however I'm no expert).
To refine this, make a Linear classifier. Train it manually: pick out anomalous 10 ? out of 100 ? waves by hand and eye, and make a linear classifier with e.g. scikit-learn to separate the 10 from the 90 / 10k from 90k.
Added 1 Dec: Spectral flatness is a measure of how noise-like a signal is; it's easy to implement, and may be adequate for your task.
(I'd suggest asking over on dsp.stackexchange.com, with tag Matlab or Python too.
Moderators, move the question ?)