Closed form for KL divergence equation

Question

For a fixed $p\in(0,1)$ and $y>0$ is it possible to find a closed form for $x\in(0,1)$ such that $$x\log\left(\frac{x}p{}\right) + (1-x)\log\left(\frac{1-x}{1-p}\right) = y ?$$ I've tried a bunch of calculations but didn't come up with anything, so any hint or help will be very useful!

Answer 1

I am skeptical about a possible closed form but, beside numerical methods, I think that decent approximations could be obtained.

Conside that you look for the zeros of function $$f(x)=x\log\left(\frac{x}p{}\right) + (1-x)\log\left(\frac{1-x}{1-p}\right) - y $$ for which $$f'(x)=\log \left(\frac{x}{p}\right)-\log \left(\frac{1-x}{1-p}\right)\qquad \text{and} \qquad f''(x)=\frac{1}{x}+\frac{1}{1-x} \quad >0$$

The first derivative cancels at $$x_*=p \implies f(x_*)=-y$$ For a first approximation, expand $f(x)$ as a Taylor series around $x_*$; this will give $$f(x)=-y+\frac{(x-p)^2}{2 (1-p) p}+O\left((x-p)^3\right)$$ giving $$x_0=p\pm\sqrt{2 p(1-p) y }$$ We also have that $$f(0)=-y-\log(1-p)\qquad \text{and} \qquad f(1)=-y-\log(p)$$ so, depending on the respective values of $p$ and $y$, we can have $0$, $1$ or $2$ solutions.

Suppose that $f(0) < 0$ leading to a single root. We have that $$y=\sum_{n=2}^\infty \frac{\left((-1)^n (1-p) p\right)^{-n} \left((-1)^n (1-p) p^n+p (1-p)^n\right) } {n(n-1) } (x-p)^n$$ Truncating to some order and using series reversion $$x \sim p+t +\frac{1-2 p }{6 [p(1-p)]}t^2-\frac{2p(1-p)+1 }{72 [p(1-p)]^2}t^3+ O(t^4)$$ where $t=\sqrt{2 p(1-p) y }$.

Trying for $p=\frac 13$ and $y=\frac 12$, this would first give $x_0=\frac{1+\sqrt{2}}{3}=0.804738$, the truncated series $x=\frac{168+131 \sqrt{2}}{432}=0.817736$ while the solution (given using Newton method) is $x=0.818888$.

All of this could be signifiantly improved at the price of a few more terms in the series expansion.