What exactly is $X^2$ and what is $E[X^2]$?

Question

I am very new to Probability theory and there are a few things that really confuse me.
Suppose $M$ is a finite Subset of $\mathbb{N}$ and $X:\Omega \rightarrow M$ a random variable.

My first question is: What is the Image of $X^2$ ? Because it cannot be $M$. So is it just all values in $M$ squared?

My second question is : why is $E[X^2] = \sum_{x \in M} x^2*P(X=x)$ and not $\sum_{x \in M} x^2*P(X^2=x^2)$ ? That is what I get when following the definition of $E[X]$.

The definition for E[X] for finite Random variables I use is:
$E[X]=\sum_{x \in M} x*P(X=x)$

Answer 1

Actually, the expression $\mathbb{E}(X^2) = \sum_{x}x^2 \mathbb{P}(X = x)$ is not directly defined to be in this way. It is indeed given by the law of the unconscious statistician, which states that,

if $X$ is a r.v. and $g:\mathbb{R} \to \mathbb{R}$, then $$\mathbb{E}(g(X)) = \sum_{x}g(x) \mathbb{P}(X = x)$$ whenever this sum is absolutely convergent.

By assigning $g$ to be $g(x) = x^2$, one can see $$\mathbb{E}(X^2) = \mathbb{E}( g(X) ) = \sum_{x}g(x) \mathbb{P}(X = x)=\sum_{x}x^2 \mathbb{P}(X = x).$$

You can find more details in 'Probability and Random Processes' by Grimmett and Stirzaker.

Answer 2

For your first question, note that the image of $X$ isn't necessarily $M$. Formally, $X$ is just a function, so its image is defined as $\{X(\omega):\omega\in\Omega\}$ which in general will be a subset of $M$. Similarly, $X^2$ is just a function with image $\{X^2(\omega):\omega\in\Omega\}$. This will be a subset of $M^2$, defined as the set of squares of elements of $M$.

You may be asking because your definition of expectation maybe requires that you sum over all values in the image of a random variable. More generally, you can sum over any set that contains the image. Any "unused" values will have zero probability and won't contribute to the summation. So, you could sum over the actual image, or the possibly larger set $M^2$ or the even larger set $\mathbb{N}$. It's all the same.

For your second question, the key is that in your definition of $E[X]$, you must sum over the unique values of a random variable and weight the terms by the probabilities for that SAME RANDOM VARIABLE to be equal to those unique values you're summing over. So: $$E[X^2] = \sum_{x\in M} x^2\,P(X=x)$$ works because you are summing over the unique values $x\in M$ of the random variable $X$ and weighting the terms by the probabilities for that random variable $X$ to be equal to $x$.

It would also work to sum over the unique values $y\in M^2$ of the random variable $X^2$ and weight the terms by the probabilities for that random variable $X^2$ to be equal to $y$: $$E[X^2] = \sum_{y\in M^2} y\,P(X^2=y)$$

However, the following does NOT work (in general): $$E[X^2] = \sum_{x\in M} x^2\,P(X^2=x^2)$$ because you are summing over the unique values of the random variable $X$ but then weighting by the probabilities that a different random variable $X^2$ takes on values that are some function of $x$.

Now, in your particular case, it just happens to be true that $$E(X^2) = \sum_{x\in M} x^2\, P(X^2=x^2)$$ but ONLY because the values in $M\subseteq \mathbb{N}$ are positive and so squaring is a one-to-one operation. (In other words, you have $P(X^2=x^2)=P(X=x)$ for all $x\in M$, so your incorrect expression is numerically equal to the first correct version given above.)

But, if you had a more general $M\subseteq \mathbb{Z}$ that could take on negative values, you might end up double counting. For example, consider the random variable $X\colon\Omega \to \{-1,1\}$ that takes on each value with probability 1/2. Since $X^2$ is always 1, it's obvious that we must have $E(X^2)=1$, and our "correct" versions of the expectation give this answer: $$E(X^2) = \sum_{x\in M} x^2\, P(X=x) = (-1)^2 P(X=-1) + 1^2 P(X=1) = 1/2 + 1/2 =1$$ and $$E(X^2) = \sum_{y\in M^2} y\, P(X^2=y) = 1 P(X^2=1) = 1$$ but the incorrect one double counts: $$\begin{align*} E(X^2) &= \sum_{x\in M} x^2\, P(X^2=x^2) = 1^2 P(X^2=1^2) + (-1)^2 P(X^2=(-1)^2) \\ &= P(X^2=1) + P(X^2=1) = 1 + 1 = 2 \end{align*}$$