I am collecting performance data for an application, I gather results each version.
Given that the measurement is imprecise, I'd like to calculate a confidence interval for the performance data.
Issue 1:
With multiple measurements I can apply a CI with a t distribution, as recommended for sample sizes less than 30. However my sample size is actually 2-4, is that too small?
Issue 2:
Many builds only run the performance tests once. How could I estimate error for that? I mean, technically there's an infinite error. But that is less than useful.
Honestly, I can't remember much from my statistics days. I'd simply like to graph an upper and lower bound, and the average.