I have two deterministic algorithms, Algorithm 1 and Algorithm 2. The first has $m$ inputs and one output, and the second has $n$ inputs and one output. The distributions of the inputs of the algorithms are assumed to be uniform distributions on $[0, 1]^m$ and $[0, 1]^n$, respectively.
Algorithm 1 is the ultimate truth, and Algorithm 2 is an approximation to Algorithm 1. I would like to study how close the distributions of the outputs of the algorithms are. To this end, I have decided to use Monte Carlo (MC) sampling and the Kolmogorov–Smirnov (KS) test. So, I draw 10000 or so independent samples from the distributions of the inputs, feed the samples into the algorithms, and apply the KS test to the resulting data.
I have a number of questions in this regard.
Question 1: How sensible it is to discard the binary answer of the KS test and to focus solely on the p-value reported by the test as a measure of the quality of the approximation of Algorithm 1 by Algorithm 2? I would like to have a metric that I can improve gradually (by adjusting Algorithm 2) instead of staring at “Reject” or “Failed to reject.” Can the p-value stand on its own and serve as such a metric?
Question 2: Since it is about computer simulation, I can easily generate new sets of samples and apply the KS test over and over again. I am concerned with the consistency of the results that I am getting. More specifically, sometimes it happens that the p-value is very small, and the null hypothesis is firmly rejected. Other times the p-value turns out to be rather large, and the null hypothesis cannot be reject. I understand that MC sampling is behind the scene, and one should expect some fluctuations. However, I am wondering if there is anything one can do to get more consistent results. For example, does it make sense to average a set of p-values computed for a range of datasets?
I would appreciate any ideas, suggestions, or any other kind of thoughts regarding what I am doing. Thank you!
Regards, Ivan