Intuition of Negative entropy as a Fenchel conjugate of logistic cost function

361 Views Asked by At

Let us first recall the logistic cost function

$$f_1(x)=\log(1+e^x)$$

and the negative entropy function

$$f_2(y)=y\log(y)+(1-y)\log(1-y).$$

Here $\log$ denotes the natural logarithm and $x\in \mathbb R$ and $y\in (0,1)$.

We also recall the Fenchel conjugate function $f^*$ of function $f$ as

$$f^*(y):=\sup_{x\in \mathbb R} \, \, yx - f(x).$$

According to Wiki, we know that $$f_1^*(y)=f_2(y).$$

The notion of logistic cost and negative entropy have their own meaning and seem to be independent. I am so surprising when the Fenchel conjugate can establish their relation! Is there an intuition behind such relation?