I'm asked to compare the distance of the approximated distribution and the true distribution in a Bayesian approach. While we used Laplace approximation to find the MAP of the target posterior distribution, one paper reviewer asked us to estimate the distance of our approximated distribution to the true posterior.
I'm confused about it. We meant to using LA to calculate an approximated MAP, so we only care about the argmax of the posterior. In our problem, the dimensionality is high and apparently a Gaussian approximation is very different to the true distribution. The overall distance will be high. So what's the point to compare the distance of the whole distribution? Plus I've been reading some machine learning papers and I haven't seen any graphical model methods, or Bayesian approaches comparing their approximated inference to an intractable exact inference.
Could anybody give me some idea about this problem? May I have a reference to see how we shall address this issue? Thanks!
Okay I suggest you to use the Hellinger distance or the squared Hellinger distance. It is symmetric and scales in $[0,1]$. You can simply talk about a certain percentage of closeness, then. Check wiki for details. If you want to have a complete list of distances you can find it here