Posterior of a corrupted transformation

Question

Let $z\sim\mathcal{N}(z|0,1)$ and let $p(x|z)=\mathcal{N}(x|f(z),\gamma)$ where $f:\mathbb{R}\to\mathbb{R}$ is a differentiable bijection. I am trying to approximate the posterior $p(z|x)$ with a distribution $q(z|x)$ in the limiting case of $\gamma\to 0$. Intuitively, it should be concentrated around $z=f^{-1}(x)$ and my calculations suggest that the respective variance is probably $\gamma/f'(f^{-1}(x))^2$, but is there a formal way to prove this ? Can we prove, for example, that the KL divergence between $q(z|x)=\mathcal{N}\bigg(z\bigg|f^{-1}(x), \frac{\gamma}{f'(f^{-1}(x))^2}\bigg)$ and $p(z|x)$ goes to zero as $\gamma \to 0$ ?

Answer 1

It might be of interest to see what the, so called, saddle point approximation (or Laplace's method) suggests as mean and variance for the posterior.

For sufficiently small $\gamma$, the posterior density will be unimodal and highly concetrated around its peak, denoted as $\hat{z}$. This suggests the series representation of the logarithm of the posterior

$$ \log p(z|x) = \log p(\hat{z}|x) - (z-\hat{z})^2\frac{c}{2} + \mathcal{O}((z-\hat{z})^3), $$

where

$$ c \triangleq -\left. \frac{\partial^2}{\partial z^2}\log p(z|x) \right|_{z=\hat{z}}. $$

One could approximate $\log p(z|x)$ by the first two terms of the above expansion, which corresponds to a Gaussian distribution with mean $\hat{z}$ and variance $1/c$.

The mean equals the maximum of $p(z|x)$ (equivalently, $\log p(z|x)$), which is easily shown to be the solution of

$$ \tag{1} \dot{f}(\hat{z})(x-f(\hat{z})) = \hat{z} \gamma, $$

whereas the variance is easily shown to be equal to

$$ \tag{2} \frac{1}{c}=\frac{\gamma}{\gamma + \dot{f}^2(\hat{z})+\gamma \hat{z} \ddot{f}(\hat{z})/\dot{f}(\hat{z})} $$

Interestingly for $\gamma \rightarrow 0$, (1) suggests $\hat{z} \approx f^{-1}(x)$ and (2) suggests $1/c \approx \gamma / \dot{f}^2( f^{-1}(x))$, as your computations.