I understand that the population variance of a set is just a special case of the variance of a random variable where $P(X=x) = \frac{1}{n}$ for $n$ elements in the set.
Still I can't help but feel that when computing $\sigma^2 (X) = \sum_{i=1}^n [(x_i -\mu)^2 \cdot P(X=x_i)]$, we're doing extra, or somehow "too much", individual weighing of the values of $X$.
After all, aren't the weights by likelihood of occurrence of a certain $x_i$ already expressed in $\mu$? Why are we again weighing the squared differences from the mean by probability? Why not average using $\frac{1}{n}$ instead?
The squared variation of $X$ from its mean could itself be considered a random variable, say defined by $Y=(X-\mu)^2$. The variation then measures the mean of $Y$, which involves using $Y$'s and hence $X$'s probability distribution. Your feelings wouldn't say to average a random variable (like $Y$) by simply adding the possible values it could take and dividing by the count if $Y$ is not uniformly distributed would they? If $x_1$ and $x_2$ are two values that $X$ can take with some probability, then the squared variation $(X-\mu)^2$ is not equally likely to be $(x_1-\mu)^2$ as it is to be $(x_2-\mu)^2$ unless for instance $X$ itself is equally likely to be $x_1$ as it is to be $x_2$.
Consider for instance a random variable $X$ that took the values $a$ or $b$ with probabilities $1$ and $0$ respectively. So essentially, $X$ always takes the value $a$, and in particular its mean is $\mu=a$.
According to your feelings, we shouldn't take into account how likely it is that $X$ is $a$ versus $b$ and simply calculate the average squared variation to be
$$\frac{1}{2}(a-a)^2+\frac{1}{2}(b-a)^2=\frac{(b-a)^2}{2}.$$
In other words, the average squared variation of $X$ from its mean would be $(b-a)^2/2$, even though $X$ never varies from its mean value $\mu=a$ and the value $b$ is completely independent of $X$ anyway, which is absurd. The issue is that the squared variation $(X-\mu)^2$ is equal to $(a-a)^2$ one-hundred percent of the time (not $\frac{1}{2}$ of the time) and is equal to $(b-a)^2$ zero percent of the time (not $\frac{1}{2}$ of the time), so on average $(X-a)^2$ should be
$$1(a-a)^2+0(b-a)^2=0.$$