What does it mean that a distribution is hard to sample from?

Question

I'm currently studying about sampling methods, and some terminology that has arisen is that a distribution is "hard to sample from".

What does this mean?

My current intuition is that the distribution is joint and we need to collect a lot of values to get a sample, or that for example the variables of the distribution are in a very high dimension, and you need to sample A LOT of data to understand how it looks like. Is this correct?

Answer 1

Good question. Very foundational to learning the true value of efficient sampling techniques.

It boils down to the simple fact that to "sample from a distribution" you need to know the CDF of the distribution. More precisely you need the Inverse CDF i.e. a function that will give you the a sample if you feed it a probability. Although all CDFs are monotonically increasing and hence theoretically invertible analytically (practically) it not that easy.

So two issues prevent you from directly getting values from the CDF. One is getting the CDF itself and the second is inverting it. The problem gets more complex in higher dimensions. So efficient sampling techniques are used to approximate the functions instead.

Hope this helps.