How Can ı validate my Monte Carlo Simulation?

Question

Now I am writing my thesis The topic is about error analysis on turbocharger test bench. In order to estimate uncertainties at test bench first of all I used Monte Carlo Method. Now I have to validate my results. I do not want to use Gauss Method. Do you have any idea how can I validate my results? Is adaptive Monte carlo method such a method to validate ? Thanks a lot.

Answer 1

The comments are correct that there are various paths and that it would be helpful if you could give more information. However, here are some generic investigations that I have found useful. They might not be the end of your 'validation', but they may raise issues that point towards the end.

(1) If you have $m = 10^6$ independent iterations in a Monte Carlo simulation of a mean or proportion, then it is reasonable to assume that mean or proportion is nearly normal. So find the relevant SD and use $2SD/\sqrt{m}$ to give a serviceable estimate of the margin simulation error.

(2) If you have questions about independence (as in Markov Chain Monte Carlo, for example) then make an ACF plot for lags out to about 40, and see how long it takes for dependence to decay. Then 'thin' the results; for instance if ACF for lag 10 and beyond is essentially 0, then use every 10th observation. Then adjust m in step (1) accordingly.

(3) If you have questions about the stability of the simulation, then look at 'traces' of running averages (cumulative sums up to n divided by n) against n. They may be very rough-looking as the simulation begins, but they should be quite smooth near the end.

(4) In the case of dependence there is always the risk that the simulation is 'getting stuck' for periods of time (or forever). A 'history plot' of the observations against time will help you to see any serious problems along these lines.

(5) If you are generating several quantities that ought to be stochastically or functionally independent, make scatter plots of pairs of such quantities (along with Pearson and Spearman correlations) to see if there are unexpected relationships. If there are many such quantities, you might try making 'matrix plots' of 'pairs' of variables.

(6) Make histograms of individual simulated quantities. Don't just look at means (or medians) and standard deviations (or interquartile ranges). A histogram of a random quantity should have an 'envelope' or 'density estimator' that is nearly the same as the distribution. If you see unexpected evidence of strong multiple modes, you should try to figure out why.

(7) Sometimes an easily computed by-product of a simulation is a quantity that is has a known value or distribution. Compute them even if they are not part of the reason for doing the simulation. If they do not have the anticipated values or approximate the anticipated distributions, that is a sign of trouble. It is usually better to compare empirical cumulative distribution functions (ECDFs) with known CDFs of distributions than to compare histograms with known PDFs (no loss of acuity due to binning with ECDFs).

I doubt that all of these suggestions will be applicable to any one simulation, but give serious consideration to these and other numerical and graphical descriptions of the output of your simulation. You may find unexpected glitches in the simulation program that are not hard to fix. You may also see that everything is working wonderfully well, and then you will feel much better about the usefulness of your simulation.

Most of the words I have put in 'single quotes' have some chance of fetching useful information in an Internet search if you need further information. Also, when you know a little more about some of these methods and about how your simulation works, you might try posting more specific and technical follow-up questions on our sister 'statistics' site.