Known inequalities between entropy and the log absolute value of the determinant?

120 Views Asked by At

Let $A$ denote an $n$-by-$n$ Hermitian positive semidefinite matrix of trace $1$. If its eigenvalues are $\lambda_i$, for $1 \leq i \leq n$, then the entropy of $A$ is defined by:

$$H(A) = - \sum_{i=1}^n \lambda_i \log(\lambda_i).$$

What are known ("universal") upper and lower bounds of the entropy $H(A)$ in terms of $\log|\det(A)|$?

2

There are 2 best solutions below

13
On BEST ANSWER

Here is an upper bound. Let $S_n$ denote the set of permutations on $\{1,2,\ldots,n\}$. For a permutation $\sigma\in S_n$, prove that $$\sum_{i=1}^n\,\lambda_i\,\ln\left(\lambda_{\sigma(i)}\right)\leq \sum_{i=1}^n\,\lambda_i\,\ln\left(\lambda_i\right)\,,$$ using the Rearrangement Inequality. Therefore, $$\sum_{\sigma\in S_n}\,\sum_{i=1}^n\,\lambda_i\,\ln\left(\lambda_{\sigma(i)}\right)\leq n!\,\sum_{i=1}^n\,\lambda_i\,\ln(\lambda_i)=-n!\,H(A)\,.$$ Ergo, $$H(A)\leq-\frac{1}{n!}\,\sum_{i=1}^n\,\lambda_i\,\ln\left(\prod_{\sigma\in S_n}\,\lambda_{\sigma(i)}\right)=-\frac{1}{n!}\,\sum_{i=1}^n\,\lambda_i\,\ln\left(\prod_{j=1}^n\,\lambda_j^{(n-1)!}\right)\,.$$ This means $$H(A)\leq -\frac{1}{n}\,\sum_{i=1}^n\,\lambda_i\,\ln\big(\det(A)\big)=-\frac{1}{n}\,\ln\big(\det(A)\big)\,.$$ The equality holds if and only if $\lambda_1=\lambda_2=\ldots=\lambda_n=\dfrac{1}{n}$.

Frankly, this inequality is weak because we have $$H(A)\leq \ln(n)\leq -\frac{1}{n}\,\ln\big(\det(A)\big)\,.$$ The inequality $\ln(n)\leq -\frac{1}{n}\,\ln\big(\det(A)\big)$ is due to the AM-GM Inequality $$\left(\prod_{i=1}^n\,\lambda_i\right)^{\frac{1}{n}}\leq \frac{1}{n}\,\sum_{i=1}^n\,\lambda_i=\frac{1}{n}\,.$$ The inequality $H(A)\leq \ln(n)$ is due to the Weighted AM-GM Inequality $$\exp\big(H(A)\big)=\prod_{i=1}^n\,\left(\frac{1}{\lambda_i}\right)^{\lambda_i}\leq \sum_{i=1}^n\,\lambda_i\,\left(\frac{1}{\lambda_i}\right)\leq n\,.$$

For the lower bound, we shall prove that $$H(A)\geq \ln(n)\,\det(A)\,.$$ Without loss of generality, suppose that $\lambda_1\leq \lambda_2\leq \ldots\leq \lambda_n\leq 1$. This implies $\lambda_1\leq \dfrac{1}{n}$. Therefore, $$H(A)=\sum_{i=1}^n\,\lambda_i\,\ln\left(\frac{1}{\lambda_i}\right)\geq \lambda_1\,\ln\left(\frac{1}{\lambda_1}\right)\geq \ln(n)\,\lambda_1\geq\ln(n)\, \lambda_1\lambda_2\cdots\lambda_n\,.$$ Thus, $$H(A)\geq \ln(n)\,\det(A)=\ln(n)\,\exp\Big(\ln\big(\det(A)\big)\Big)\,.$$ Note that the equality holds if and only if there exists $i\in\{1,2,\ldots,n\}$ such that $$\lambda_1=\lambda_2=\ldots=\lambda_{i-1}=\lambda_{i+1}=\ldots=\lambda_{n-1}=\lambda_n=0$$ and $$\lambda_i=1\,.$$ (However, you also have $\lambda_1\leq \big(\det(A)\big)^{\frac{1}{n}}$. Therefore, you can also write $$H(A)\geq -\frac{1}{n}\,\lambda_1\,\ln\big(\det(A)\big)\geq -\frac{1}{n}\,\det(A)\,\ln\big(\det(A)\big)\,.$$ This is a slightly better lower bound.)

In conclusion, we have $$\ln(n)\,\exp\Big(\ln\big(\det(A)\big)\Big)=\ln(n)\,\det(A)\leq H(A)\leq \ln(n)\leq -\frac{1}{n}\,\ln\big(\det(A)\big)\,.$$ If you do not want $n$ in the expression for the lower bound, you can use something like $\ln(n)>1$ for $n\geq 3$, and get $$H(A)\geq \det(A)$$ for $n\geq 3$. For the upper bound, you have $-\ln\big(\det(A)\big)\geq 0$, so $$H(A)\leq -\ln\big(\det(A)\big)\,.$$ Hence, when $n\geq 3$, we have $$\det(A)\leq H(A)\leq -\ln\big(\det(A)\big)\,.$$

0
On

Let $n\geq 2$. The upper bound of $\det(A)$ is $\min\left\{\dfrac{H(A)}{\ln(n)},\dfrac{1}{n^n}\right\}$ as shown in the previous answer. To get a lower bound of $\det(A)$ in terms of $n$ and $H(A)$, we note that $$\ln\big(\det(A)\big)=\sum_{i=1}^n\,\ln(\lambda_i)\,.$$ With the assumption that $\lambda_1\leq\lambda_2\leq\ldots\leq\lambda_n\leq 1$, we see that $$\lambda_1\,\ln\big(\det(A)\big)=\sum_{i=1}^n\,\lambda_1\,\ln(\lambda_i)\geq \sum_{i=1}^n\,\lambda_i\,\ln(\lambda_i)=-H(A)\,.\tag{$\star$}$$ Rewrite $H(A)$ as $$H(A)=-\left(\sum_{j=2}^n\,\lambda_j\right)\,\sum_{i=1}^{n-1}\,\mu_i\,\ln\left(\mu_{i}\right)-\left(\sum_{j=2}^n\,\lambda_j\right)\,\ln\left(\sum_{j=2}^n\,\lambda_j\right)-\lambda_1\,\ln(\lambda_1)\,,$$ where $\mu_i:=\dfrac{\lambda_{i+1}}{\sum\limits_{j=2}^n\,\lambda_j}$ for $i=1,2,\ldots,n-1$. Since $\sum\limits_{j=2}^n\,\lambda_j=1-\lambda_1$, $$H(A)=-(1-\lambda_1)\,\sum_{i=1}^{n-1}\,\mu_i\,\ln(\mu_i)+h(\lambda_1)\,,$$ where $$h(t):=-t\,\ln(t)-(1-t)\,\ln(1-t)$$ for $t\in[0,1]$.

We know that $-\sum\limits_{i=1}^{n-1}\,\mu_i\,\ln(\mu_i)\leq \ln(n-1)$ from my previous answer. Therefore, $$h(\lambda_1)=H(A)-(1-\lambda_1)\,\left(-\sum_{i=1}^{n-1}\,\mu_i\,\ln(\mu_i)\right)\geq H(A)-(1-\lambda_1)\,\ln(n-1)\,.$$ In other words, $$-\lambda_1\,\ln(\lambda_1)-\left(1-\lambda_1\right)\,\ln(1-\lambda_1)-\lambda_1\,\ln(n-1)\geq H(A)-\ln(n-1)\,.$$ Since $\lambda_1\leq \dfrac{1}{n}$ and the function $\eta:[0,1]\to\mathbb{R}$ defined by $$\eta(t):=-t\,\ln(t)-(1-t)\,\ln(1-t)-t\,\ln(n-1)$$ for $t\in[0,1]$ is concave, we can see that $$\eta(t)\leq \eta'\left(\frac{1}{n+1}\right)\,\left(t-\frac{1}{n+1}\right)+\eta\left(\frac{1}{n+1}\right)\,,$$ or $$\eta(t)\leq \ln\left(\frac{n}{n-1}\right)\,t+\ln\left(\frac{n+1}{n}\right)$$ for all $t\in[0,1]$. Consequently, $\eta(\lambda_1)\geq H(A)-\ln(n-1)$ implies that $$\lambda_1\geq\max\left\{\frac{H(A)-\ln\left(\frac{n^2-1}{n}\right)}{\ln\left(\frac{n}{n-1}\right)},0\right\}\,.\tag{$\Box$}$$

If $H(A)>\ln\left(\frac{n^2-1}{n}\right)$, we use ($\star$) and ($\Box$), along with the fact that $\ln\big(\det(A)\big)\leq 0$, to obtain $$\left(\frac{H(A)-\ln\left(\frac{n^2-1}{n}\right)}{\ln\left(\frac{n}{n-1}\right)}\right)\,\ln\big(\det(A)\big)\geq -H(A)\,.$$ Thus, $$\det(A)\geq \exp\left(-\frac{\ln\left(\frac{n}{n-1}\right)\,H(A)}{H(A)-\ln\left(\frac{n^2-1}{n}\right)}\right)$$ if $H(A)>\ln\left(\frac{n^2-1}{n}\right)$.

Of course, you can try to use this inequality instead to get a different bound. Let $\epsilon>0$. We have $$\eta(t)\leq \eta'\left(\frac{1}{n+\epsilon}\right)\,\left(t-\frac{1}{n+\epsilon}\right)+\eta\left(\frac{1}{n+\epsilon}\right)\,,$$ or $$\eta(t)\leq \ln\left(\frac{n-1+\epsilon}{n-1}\right)\,t+\ln(n+\epsilon)-\ln(n-1+\epsilon)\,.$$ This gives $$\lambda_1\geq \max\left\{\frac{H(A)-\ln\left(\frac{(n-1)(n+\epsilon)}{n-1+\epsilon}\right)}{\ln\left(\frac{n-1+\epsilon}{n-1}\right)},0\right\}\,.$$ If $H(A)>\ln\left(\frac{(n-1)(n+\epsilon)}{n-1+\epsilon}\right)$, then $$\det(A)\geq \exp\left(-\frac{\ln\left(\frac{n-1+\epsilon}{n-1}\right)\,H(A)}{H(A)-\ln\left(\frac{(n-1)(n+\epsilon)}{n-1+\epsilon}\right)}\right)\,.$$

As mentioned in a comment under my first answer, if $0\leq H(A)\leq \ln(n-1)$, then it is possible that $\det(A)=0$, so $0$ is the only lower bound for $\det(A)$ in this case. However, if $\delta:=H(A)-\ln(n-1)$ is positive, then by taking $\epsilon:=\frac{1}{\delta}$, the condition $H(A)>\ln\left(\frac{(n-1)(n+\epsilon)}{n-1+\epsilon}\right)$ is met. Ergo, for $H(A)>\ln(n-1)$, we have an even more complicated lower bound: $$\begin{align}\det(A)&\geq \exp\left(-\frac{\ln\left(\frac{1+(n-1)\,\big(H(A)-\ln(n-1)\big)}{(n-1)\,\big(H(A)-\ln(n-1)\big)}\right)\,H(A)}{\big(H(A)-\ln(n-1)\big)-\ln\left(\frac{1+n\,\big(H(A)-\ln(n-1)\big)}{1+(n-1)\,\big(H(A)-\ln(n-1)\big)}\right)}\right) \\ &{\color{red}\geq}\Big((n-1)\,\big(H(A)-\ln(n-1)\big)\Big)^{\frac{2\,H(A)}{(2n-1)\,\big(H(A)-\ln(n-1)\big)^2}} \\&\ \ \ \ \,\exp\left(-\frac{2(n-1)\,H(A)}{(2n-1)\,\big(H(A)-\ln(n-1)\big)}\right)\\ &\ \ \ \ \ \ \ \ \ \ \exp\left(-\frac{(n-1)(6n^2-3n+1)\,H(A)}{3(2n-1)^2}\right)\,.\end{align}$$ This looks awful. (I am not very certain about the inequality in ${\color{red}{\text{red}}}$.)