In this paper Boltzmann Machines, the first paragraph of the second section 2. Boltzmann Machines argues that,
we could never train such a network with three units to visit the states (0,0,0), (0,1,1), (1,0,1) and (1,1,0) with some probabilities, but not the other four possible states. This is because the first and second order statistics (the mean of each component and correlations between components) are the same for these four states as the remaining four and so the network can not discriminate them.
As I learned from here, order logic is just to order the attributes of the samples and take the i-th element of it.
I am wondering why the first and second order statistics of the two sets are the same.
Thank you!
On the surface, it seems like the point they're trying to make has more to do with identifiability than order statistics. Perhaps, in their language, "order statistics" refers to moments (in statistics) or powers of a sample. This is evidenced by their statement in parentheses about the mean and covariance.
If they are actually referring to moments, and a finite set of moments drive the states in question (e.g. as a normal distribution is fully determined by its mean and [co]variance), then they might simply be describing the model's inherent symmetry and/or limitations by way of its low-order moments.
Also, it sounds like they're voicing concerns similar to those that arise when considering the identifiability of mixture models.