Inverse Transform Theorem

Question

I am taking an optional stats graduate course (without prior stats experience) and I have difficulties in understanding basics.

I understand concept of inverse function, e.g. if we have f(2) = 5 then we plug 5 into inverse function of f(x) and we get 2 as a result. I am also familiar with concepts of p.d.f and c.d.f, as well as normal, exponential, uniform etc distributions.

I do not understand a concept of "Inverse Transform Theorem" which states that:

"Let X be a continuous random variable with c.d.f. F(x). Then F(X) is approximately distributed uniformly U(0,1)."

How is it possible that for every F(X) (exponential, normal...) we get uniform distribution U(0,1)? Some visualisation would be really helpful to understand what is going on. I tried to search for the answer using Google and Youtube and there are a lot of resources but still but couldn't understand this (probably simple) concept, which is nowhere explained.

I am also having troubles understanding the difference between F(x) and F(X). If big X is a continuous random variable, then what is small x?

Edit:

As you can see on the graph below, it is easy to understand that in uniform distribution, each random variable X between a and b has equal chance of being selected. So, I do not understand how this applies to Inverse Transform Theorem... how below graph fits on the Y-axis where inverse function outputs values out of normal or exponential input values? Why output of inverse function is uniform for any input?

An example of Inverse Transform Theorem:

Answer 1

I think I got it. Because input into inverse function in random, hence each value on the Y-axis (output of inverse function) has equal chance of occurence, thus it has uniform distribution.

Ahhhh... so simple. :) I hope I am right.

Answer 2

Now, how about choosing 10,000 observations x at random from $\mathsf{Exp}(1),$ and then checking whether $U_i = 1 - e^{-X_i}$ have a standard uniform distribution. [Sampling and computations in R, where rexp generates exponential data, pexp is an exponential CDF, and dunif is a uniform density function.]

set.seed(2020)
x = rexp(10000, 1)
summary(x)
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
0.000148 0.296723 0.709628 1.014589 1.387810 9.063560 

u = pexp(x)
summary(u)
     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
0.0001484 0.2567498 0.5081731 0.5055758 0.7503787 0.9998842 

hist(u, prob=T, col="skyblue2")
 curve(dunif(x,0,1), add=T, col="red", n=10001)

Note: This is an important idea. The reverse procedure allows you to get a random sample from any distribution for which you know the inverse of the CDF (quantile function). Thus, starting with a random sample from $\mathsf{Unif}(0,1),$ you can use the exponential quantile function to get your random sample: So

set.seed(1023)  # for reproducibility
y = qexp(runif(1000, 0,1), 1)

will get you a random sample of size $n = 1000$ from an exponential population with rate 1 (and hence mean 1). [Note that the quantile function qexp is $F_Y^{-1}(u) = -\ln(1-u).]$

You can use a Kolmogorov-Smirnov goodness-of-fit test to see if observations y are consistent with a sample from $\mathsf{Exp}(1).$ [P-value > 0.05 indicates so.]

ks.test(y, pexp, 1)

       One-sample Kolmogorov-Smirnov test

data:  y
D = 0.013835, p-value = 0.9909       
alternative hypothesis: two-sided

hist(y, prob=T, col="skyblue2", main="Sample from EXP(1)")
 curve(dexp(x,1), add=T, col="red")