probability variance formula intuition

Question

Here is the formula for the variance : $\sigma^2=\dfrac{\sum(X-\mu)^2}{N}$.

My question is why do we SQUARE the difference between the mean and the variable, why don't we use absolute value ...The result will be different!

I saw online that it's to make the value positive, but like why don't we choose 4 or 6 as exponent?

Hope someone can enlighten me!

Thanks :) Ayman

Answer 1

The expected distance from the mean $E\left(\left|X-E(X)\right|\right)$ is also a valid measure of the variability of a random variable. This is known as mean absolute deviation or MAD.

The expected squared distance from the mean, the variance $E\left[\left(X-E(X)\right)^2\right]$ has established itself historically because it is easier to handle analytically while it still stays positive. As Jimmy R. mentioned you can still differentiate after squaring while the same is not true for taking the absolute value.

Another property you should be aware of is that since squaring is a convex transformation, larger deviations from the mean affect the variance more strongly than smaller deviations from the mean. The same is not true for MAD.

Answer 2

Here's a reason (not necessarily the reason). You can use either of the following measures of deviation from a given point $x$, i.e. how closely a set of values $x_1,...,x_n$ are grouped round a given point $x$ (smaller values being more closely):

1. $\frac1n\sum_{i=1}^n(x_i-x)^2$ or
2. $\frac1n\sum_{i=1}^n|x_i-x|$.

How would you use either of these measures to determine how closely the set of values are grouped together (i.e. if you don't yet have a specific value $x$ you're interested in, just the values $x_i$)? The natural thing to do is to say we'll work out how closely grouped around the best choice $x$ they are, i.e. the choice of $x$ which makes the deviation as small as possible. For option 1 (variance), the best value of $x$ is always the mean of the set of numbers (this is easy to check by differentiating wrt $x$). That means mean and variance naturally go together - if you wanted a measure for how close to the mean the values are, variance would be it.

But for option 2, the best value of $x$ to take is the median. So median and "average absolute deviation" go together like mean and variance. Option 2 isn't something you work with very often, simply because the median isn't used anywhere near as often as the mean.

Answer 3

Note that $f_r(x)=|x^r|$ is always convex when $r\geq 1$.

Variance or absolute deviations are nothing but Risk functions. Risk functions are expectation over location invariant convex function of random variables.

So it is not a problem to use absolute deviations or central moments of order $4$ or $6$, but when you study about risk functions, you will see a lot of derivations are easy-to-do if you take square error loss function or simply variance as a risk function!

So variance become widely used because of it's simplicity.

Answer 4

Advantages of $E[(X-\mu)^2]$ (where $\mu = E(X)$) are ease of proving theorems and established usage in estimation theory (such as UMVUE, for Uniformly Minimum Variance Unbiased Estimation). Disadvantages in some applications include the change to squared units and a possible overemphasis on extreme deviations from $\mu,$ as mentioned in the Answer by @1524 (+1). The exponent $4$ is used, but $E[(X-\mu)^4]$ is usually considered to be a measure of kurtosis (peakedness), not a measure of dispersion. [And $E[(X-\mu)^3],$ which can be negative as well as positive, is regarded as a measure of skewness.]

Advantages of $E[|X-\mu|]$ include applicability to distribution families in which $E[(X-\mu)^2]$ does not exist, maintaining the original unit scale, and robustness against outliers and contaminating effects.

The mean absolute deviation from the population median $\eta,$ $E[|X-\eta|]$ (sometimes called MAD) is also used, as are other measures of dispersion--some specifically suited for use with particular population distributions.

Answer 5

Short answer: We want a function which is differentiable everywhere.

After a concept is proposed, we will do anything we want on $it$ (so poor $it$ is T_T), like plus, minus, multiplication, integral, differentiation, and so on. Therefore, it would be better if $it$ can be done by these operations.

As we can see that the function $f(x)=|x|$ is not differentiable at $x=0$. You may image that it is not smooth at $x=0$. Therefore, there are nuisances when we consider some thing at this point!

However, the function $f(x)=x^2$ is smooth enough everywhere!

Generally speaking, a polynomial will differentiable everywhere on $\mathbb{R}$.

Yes, you can take $f(x)=x^4$ or $f(x)=x^6$, but it is not necessary.

Answer 6

Also, another benefit is that the square function punishes observations far away from the expected value much more, than the absolute value function does. So when we minimize with respect to square function instead of the absolute value function, we consider observations far away as important. The farther away, the more weight is has.