Imagine there's a guessing game where a series of binary symbols are presented and a human must decide quickly if the symbol is the same as the previous or different. There's a property of the sequence which I'm trying to define mathematically which corresponds to how difficult it is to guess whether the next symbol will be the same or different. It's partly entropy, but..
On the one hand, the sequence 010101... has maximal entropy per bit. On the other hand, it is an unchanging series of the word '01' - not very confounding. (I use the word 'confounding' because I'm not sure if it's correct to say its 'second order entropy would be zero'). How can you maximize entropy across all possible word lengths at once ? What's an optimal guessing algorithm that works well assuming that the game creator wants to use the most 'confounding' sequence ? What if you can't know the confounding-ness of the sequence up-front ?
I'm reading up on Huffman encodings but any pointers to other similar topics would be great. I'm trying to find out what exactly is this notion of 'confounding' that I'm describing in real mathematical terms ?
By subtracting $\bmod 2$, the question is equivalent to predicting the next value of the sequence. Under the assumption that the sequence is computable, you want a sequence with high Kolmogorov complexity; the higher the Kolmogorov complexity of a sequence, the harder it is for someone to even describe it, let alone guess it. The sequence 1111... (which corresponds to the sequence 010101... in your game) has very low Kolmogorov complexity, so it is very easy to guess.
The best strategy, more or less, for playing this game is Solomonoff induction, but unfortunately it's not itself computable. Solomonoff induction is a formalization of Occam's razor, which suggests that you should guess simpler hypotheses before complex hypotheses. In this context that means guessing sequences with lower Kolmogorov complexity before sequences with higher Kolmogorov complexity, and the Solomonoff prior is a particular way of doing that which has nice theoretical properties.