First of all, I am not that great of a mathematician, so please bear with me if some statements are wrong.
I have a distribution of errors (which was generated by an error function that was fed 5000 samples) that looks like this:
The function that calculated the distribution errors is now given another test sample and produces the test error.
I want to know at this point, how probable it is that the test sample was generated from the same distribution as the original samples that generated said distribution. (This seems to be an exponential distribution, right?)
For instance, if the sample error exceeds 100, it was probably generated by a faulty sample. So the corresponding probability should be low. The other way round, if I get an sample error of 10, the probability should be high.
Question: is there a way to generate probabilities for new samples?
I have read about t-tests that can do this kind of stuff, but they seem to be limited to normal distributions. Maybe there are equivalent tests for these kinds of distributions?
I would appreciate it if you keep your answers simple and understandable for me.
Thanks!
You have two samples and you want to test if they come from the same distribution.
Let $F_1 (x)$ and $F_2(x)$ be their respective empirical distribution functions. In case you have not heard of them before, they are estimates of the true distribution functions based on the sample:
$$F_1 (x) = \frac{1}{n} \sum_{i=1}^n 1\{ X_i \leq x \}$$
ie $F_1(x)$ gives the proportion of observations in sample $1$ that are less than or equal to $x$. Then the quantity:
$$\sup_{x \in \mathbb{R}} |F_1 (x) - F_2 (x)|$$
The critical value for rejection/acceptance depends on the sample size of each sample. A table of particular values is given here.
This is called the Kolmogorov-Smirnov two sample test. The KS test is a very useful statistical test and I recommend getting to know how to use it.