What is a Skewed Probability Distribution?

Question

I am currently learning about the concept of cross entropy and based on several definitions of information and entropy I came across the term skewed distribution in the following statement

... If we have a skewed distribution then we have low entropy because it is not surprising.

I am not familiar with this term except the word "skew" in linear algebra which I also don't know if its related or not. Any help is much appreciated!

Answer 1

The other answer is excellent. Here is another way to think about it, if you are only focusing on the entropy aspects.

The entropy of a probability distribution is invariant under permutations. Given the distribution $(\mathbb{P}(x): x \in X)$ where we assume $X$ is a finite set for simplicity, it doesn't matter whether $X=\{1,2,\ldots,n\}$ or $X=\{red,~green,~\ldots,~cyan\}$ the entropy $$H(X)=-\sum_{x\in X} \mathbb{P}(x) \log \mathbb{P}(x)$$is the same.

So we may define a distribution as skewed if there is a permutation $\pi$ of $X$ which results in a monotone sequence $(\mathbb{P}(\pi(x))).$

Answer 2

I am not familiar with the concept of cross entropy or anything related to it, but you can think of skewness to be the amount of asymmetry of a distribution. It is also related mathematically to the third central moment of any given distribution. In particular, we say that a distribution is right-skewed if we have a higher probability of finding values to the right of the mean as compared to the left of the mean and vice versa for a left-skewed distribution. Note that a unimodal distribution with zero skewness does not necessarily imply that this distribution is symmetric. However, the converse is true.