A couple of questions related to the following problem (please note that I am not familiar with all the theory behind Lagrange multipliers).
Starting from the definition of entropy $$H(p) = -\int p(x) \ln \left(p(x)\right) \,\mathrm d x$$ derive the general equation for the maximum-entropy distribution given constraints expressed in the general form $\int$ $b_{k}(x)p(x)dx = a_{k}$, $k= 1, 2, \dots, q$ as follows:
(a) Use Lagrange undetermined multipliers $λ_{1}, λ_{2}, ..., λ_{q}$ and derive the synthetic function: $H_{s} = -$ $\int{p(x)(ln(p(x))-\sum_{k=0}^{q}{λ_{k}b_{k}(x)})dx}-\sum_{k=0}^{q}$ ${λ_{k}a_{k}}$ State why we know $a_{0} = 1$ and $b_{0}(x) = 1$ for all $x$.
No problem with this.
(b) Take the derivative of $H_{s}$ with respect to $p(x)$. Equate the integrand to zero, and thereby prove that the minimum-entropy distribution obeys:
$p(x)=e^{\sum_{k=0}^{q}{\lambda_{k}b_{k}(x)-1}}$
where the $q + 1$ parameters are determined by the constraint equations.
2 things I do not understand:
- Why is it valid to differentiate wrt $p(x)$? How about the impact of such a differentiation on $b_{k}(x)$ (such an impact is ignored in the solution, which treats $b_{k}(x)$ as a constant)?
- Equating the integrand to $0$ is sufficient, but can we tell whether this solution is the only one (the integral might evaluate to 0 despite its integrand being different from 0)?
The method of Lagrange multipliers, when used for convex optimization, gives you a witness that what you've found is an optimum. In this case the witness can be defined pointwise. Define $\phi_x(p)=-p[\ln p-\sum_{k=0}^{q}{λ_{k}b_{k}(x)}]$ as a function $[0,\infty)\to\mathbb R.$ For each fixed $x,$ the function $\phi_x$ is strictly concave, and the particular $p(x)$ you derive in (b) obeys $\phi_x'(p(x))=0,$ so $p(x)$ is the unique maximum of $\phi_x.$ It's valid to differentiate with respect to $p$ because $x$ is fixed.
That $p(x)$ is the unique maximum means $\phi_x(q)<\phi_x(p(x))$ whenever $q\neq p(x).$ Now letting $x$ be a free variable again, it is obvious that any function $q:\mathbb R\to [0,\infty)$ satisfies $\int\phi_x(q)dx\leq \int\phi_x(p)dx,$ with equality only if $q$ is equal to $p$ almost everywhere, or assuming continuity. This is just using the fact that the integral of a non-negative function is non-negative, with equality only for functions that are almost everywhere zero.
To conclude, suppose that $q$ satisfied the same constraints as $p$ - you would find that $\int\phi_x(q)dx- \int\phi_x(p)dx=H(q)-H(p).$