Trying to understand expectation and density functions

Question

Is it accurate to say that the probability density function is the function that dictates how likely any particular value is to be taken on by a random variable? For example if $f(x)$ is the pdf, then $f(k)$ is the probability that the random variable takes on the value $k$?

For example for a single fair die, there are only 6 values to take on, so $E(x) = \sum_{k=1}^{6} x f(x)$ where $f(x)$ is the probability density function of a fair die, would it just be $f(x) = 1/6$?

And let's say it was a uniform distribution where values range anywhere from $4$ to $11$. Then it would be an integral right, because it can take on any value (not just discrete) between them? $E(x) = \int_4^{11} x f(x)dx$? But then how do I know what $f(x)$ looks like?

(I am trying to derive this intuition / figure out the correct formulas to learn better, so please don't link me to Wiki or something)

Answer 1

I will try to give you some intuitive explanations about density functions. They will not substitute for what is in your textbook, but may help you understand what is going on. First, you must make a distinction between discrete and continuous distributions.

Discrete: Let $X$ be a discrete random variable taking on a finite or countable number of values. Then, as you say in your first paragraph, the probability distribution function (or PDF or point mass function or PMF) $f_X(k) = P(X = k).$ Also, $f_X(k) \ge 0$ for all possible values $k.$ Also, $\sum_{k \in S} f_X(k) = 1,$ where $S$ is the finite or countable set of all posible values $k.%

Example: Suppose I toss a fair coin $n = 20$ times. 'Fair' means that $p = P(Heads) = 1/2$ on each toss. And tosses are independent. Then the number $X$ is 20 tosses has a binomial distribution. Then the PDF if $X$ is $f_X(k) = {20 \choose k}/2^{20},$ for $k = 0, 1, \dots, 20.$ Obviously, $f_X(k) \ge 0,$ for all $k.$ And one can show that $\sum_{i=0}^{20} f_X(k) = 1.$

If I want to compute the probability for a particular $k = 10,$ then it is $P(X = 10) = f_X(10) = {20 \choose 10}/2^{20} = 0.1762,$ computed in R statistical software as:

 dbinom(10, 20, .5)
 ## 0.1761971

Strictly speaking, probability for a discrete random variable is defined only for `points', values $k = 0, 1, \dots, 20.$ I can ask for the probability of an interval, such as $P(4 < X \le 15),$ but that is not really an interval, it is just shorthand for adding up the probabilities for the individual values $5, 6, \dots, 15$ contained in the interval. With some work or with software one can find the required sum of eleven probabilities to be 0.9882.

 sum(dbinom(5:15, 20, .5))
 ## 0.9881821

Continuous: For a continuous random variable, individual points do not matter and only intervals have probability. Suppose you are interested in heights of people. You carefully measure heights of 1000 people and make a histogram. Smoothing off the histogram, you hope to get an idea how heights are distributed in the population.

You might get a histogram like the one below. The (slightly lumpy) dotted green line is a 'density' estimator. It is a particular way of smoothing a histogram. Because experience has shown that heights are frequently normally distributed, I also included the smooth blue curve, which is the density function of a normal distribution with mean $\mu=68$ and standard deviation $\sigma =3.5.$

Either curve could serve as a density function. I'll use the normal curve because it has a formula you can find in your text and it's programmed into R.

A density function $f_X(x)$ of a continuous distribution must have three properties:

(1) $f_X(x) \ge 0.$ (You would never get a negative value smoothing a histogram; none of the bars go below the axis.)

(2) $\int_{-\infty}^\infty f_X(x)\,dx = 1.$ (The total area beneath the density function is unity. The probability that a person has some height is $1$.)

(3) $P(a < X \le b) = \int_a^b f_X(x)\,dx.$ In order to find the probability of a particular interval, integrate the density function over that interval. This is the area under the density curve between $a$ and $b$. So probability amounts to area under the density curve.

It does not make sense to ask for a probability such as $P(X = 70).$ We say that $P(X = 70) = 0.$ If we ask 'What is the probability a randomly chosen person is 70" tall, we don't mean 70.00000" tall. We ordinarily mean "What is $P(69.5 < X \le 70.5) = ?$"

That is an interval, and we can evaluate its probability. Actually, calculus doesn't work here, but there are ways to do the computation. [Use software, or standardize and use a printed normal table.] In software we get 0.0966. (In R pnorm stands for the normal cumulative distribution function: $F_X(a) = P(X \le a) = \int_{-\infty}^a f_X(x)\,dx.$)

 pnorm(70.5, 68, 3.5) - pnorm(69.5, 68, 3.5)
 ## 0.09659231

Maybe only people 67" or shorter are allowed to pilot a particular type of fighter plane with a small cockpit. What is the probability a randomly chosen person for our population would qualify according to this rule? We seek $P(X \le 67) = 0.3875.$ Or maybe, $P(X \le 67.5) = 0.4432.$

 pnorm(67, 68, 3.5)
 ## 0.3875485
 pnorm(67.5, 68, 3.5)
 $$ 0.4432015

Finally, you specifically asked about the continuous uniform distribution $Unif(4, 11).$ It's density function has a constant value $f_X(x) = 1/7$ for $x$ in $(4, 11)$, and $f_X(x) = 0$ outside of that interval. Then it satisfies rule (2) above:

$$\int_{-\infty}^\infty f_X(x)\,dx = \int_{-\infty}^4 0\,dx + \int_4^{11} (1/7) \,dx + \int_{11}^\infty 0\,dx = 0 + 1 + 0 = 1.$$

Also, for this uniform distribution, you will find that $E(X) = \int_{-\infty}^\infty xf_X(x)\,dx = 7.5.$

As requested: R code for graph of uniform PDF.

 curve(dunif(x, 4, 11), 3, 12, lwd=3, n=1001, ylab="Density", 
      main = "PDF of UNIF(4,11)", ylim = c(-.004, 1/7))
 abline(h=0, col="darkgreen")
 points(7.5, -.003, pch=17)

R code for fake normal data, histogram, density estimator, PDF curve. First line randomly simulates a sample of 1000 from NORM(68, 3.5). This will be somewhat different each time the program is run, so histogram and density estimator will differ a bit on each run.

 x = rnorm(1000, 68, 3.5)
 hist(x, br=25, prob=T, col="wheat", main="1000 Heights")
   lines(density(x), lwd=2, col="darkgreen", lty="dotted")
   curve(dnorm(x, 68, 3.5), lwd=2, col="blue", add=T)

To be fussy, the last line might have used mean and SD of the simulated sample, but with 1000 observations sample mean and SD are pretty close to population $\mu$ and $\sigma,$ so you might not notice a difference:

   a = mean(x);  s = sd(x)
   curve(dnorm(x, a, s), lwd2, col="blue", add=T)

Answer 2

Note that the probability density function, $f(x)$, of a cumulative distribution function $F(x)$, $x\in \mathbb{R}$, is defined as follows:

\begin{align} f(x)&=\lim\limits_{h\rightarrow 0}\frac{F(x+h)-F(x)}{h}\\ &=\lim\limits_{h\rightarrow 0}\frac{P((-\infty,x+h])-P((-\infty,x])}{h}, \end{align}

where we assume $P$ is the probability measure generated by $F$. In the above definition, for a discrete distribution, $f(x)$ is not defined (the limit does not exist) for all $x$ belonging to the alphabet. For all other values of $x$, $f(x)=0$.

Thus, one does not define the notion of a pdf for a discrete distribution. As pointed out in the comments, the pdf can be $\geq 1$.