Alternate definition of Entropy

Question

I am going through the book Computational Optimal Transport by Peyré and Cuturi. In it (see Formula (4.1), P. 65/209) I came across the definition of the discrete entropy, which is not what I expected: $$ H(P) = - \sum_{i,j}P_{i,j}(\log(P_{i,j})-1), $$ with $P \in [0,1]^{n,m}$ being a coupling matrix, i.e. describing the transformation of discrete propability distributions. What confuses me is the $-1$ in the sum and I would like to know, why it is included. Since $\sum_{i,j} P_{i,j}=1$ it is shifting the result by $-1$. Still it confuses me.

Answer 1

The obvious reason $-\sum_{i,\,j}P_{i,\,j}\ln P_{i,\,j}$ is a more common definition is that it's extensive, but we often don't care about that, so much as about relative entropies. In that case, one may as well add a constant $c$ to taste, giving $c-\sum_{i,\,j}P_{i,\,j}\ln P_{i,\,j}=-\sum_{i,\,j}P_{i,\,j}(\ln P_{i,\,j}-c)$. The choice $c=1$ has the unique attraction that the whole expression is just $-\sum_{i,\,j}\int_0^{P_{i,\,j}}\ln pdp$.

Answer 2

I think it might just be a strange error... In Cuturi's Paper on Sinkhorn Distances entropy is defined in the usual way.

Answer 3

Adding to the answer by @J.G, one reason for adding the -1 term could be to have cleaner expressions for the partial derivative of the dual formulation with respect to the transport plan matrix $\mathbf{P}$.

If there was no $-1$ term, then the partial derivative (page 63 of Computational Transport by Peyré and Cuturi) would have an extra $\epsilon$ term. Having the $-1$ in the coupling entropy eliminates this $\epsilon$ term, leading to a simpler expression for the Sinkhorn iterations.