According to my study, the confidence intervals for the difference in means provide a range of likely values for the difference between 2 population means. If 95% confidence interval for the difference of 2 sample means include null value, we can claim that there is no statically significant difference between the two groups.
For this comparison, do two populations need to be equivalent? Or can it be two different populations?
(Actual problem: I have test data collected from 2 test scenarios, where the settings of each test is different from each other. I want to compare the two mean values obtained from them. Note that the two distributions do not have normal shape (bell shape))
What you have described in your first paragraph assumes normality of datasets. So, if it is reasonable to make such assumption for the data collected in each of your scenarios then you can apply normal confidence interval method.