If we denote the mean as $\mu$, then the standard deviation is: $$\sigma\equiv\left(\sum_{x\in X}{p(x)(x-\mu)^2}\right)^\frac{1}{2}$$ In other words, $\sigma$ is the average $l_2$ distance from $\mu$.
In general, if we use a different metric $$\sigma_n \equiv \left(\sum_{x\in X}{p(x)|x-\mu|^n}\right)^\frac{1}{n}$$ It seems that the choice of $n=2$ is quite arbitrary, Is there a reason for that?
No, $\sigma$ is not the average $\ell^2$ distance from $\mu$. The $\ell^2$ distance from $x$ to $\mu$ is $\sqrt{(x-\mu)^2} = |x-\mu|$ and the average of those is $$ \sum_{x\in X} p(x)|x-\mu|. $$ Rather $\sigma$ is the $\ell^2$ distance from the tuple of $x$ values to the tuple in which every component is $\mu$.
A reason for the use of the mean squared deviation is that is it additive. Suppose $p$ and $q$ are probability mass functions on $X$ and $Y$ and $\mu$ and $\nu$ are averages with respect to those two measures. Then $$ \sum_{(x,y)\in X\times Y} ((x+y)-(\mu+\nu))^2p(x)q(y) = \sum_{x\in X} p(x)(x-\mu)^2 + \sum_{y\in Y} q(y) (y-\nu)^2. \tag 1 $$
Thus, suppose I toss a coin $1800$ times. The expected number of heads is $900$; the standard deviation of the number of heads is${}\,\ldots\,{}$what?? The equality $(1)$, iterated, quickly gives me the answer to that question. With that answer, I can use the central limit theorem to find the probability that the number of heads is between $890$ and $915$ inclusive.