Will Jeffrey's Prior Always be Improper?

Question

If our prior distribution has a support which ranges to infinity, then will the Jeffrey's prior necessarily be improper? For example, with a Gamma or Normal prior, the Jeffrey's prior is improper. But, with a Beta prior, we get an actual distribution when calculating the Jeffrey's prior. The support for the Gamma and Normal distributions range to infinity, whereas the support for the Beta distribution is $[0,1]$. Does this generalize to all priors whose support ranges to infinity?

EDIT: Also, if possible, it would be really interesting to see this claim proven (although I'm not sure something like that is even possible).

Answer 1

The Gaussian distribution $f(x,\theta)=\frac{1}{\sqrt{2\pi}}\exp(-(x-\mu)^2)$ has Jeffrey's prior $p(\mu)\propto 1 $, that is to say the Jeffrey's prior is the uniform distribution.

The uniform distribution on $(-\infty,\infty)$ is an improper prior.

But...

Let $f(x,\theta)=\frac{1}{\sqrt{2\pi}}\exp\left(-(x-\frac{1}{1+\theta^2})^2\right)$.

Since Jeffrey's prior is invariant under reparamaterisation, the Jeffrey's prior for this distribution is $p(\frac{1}{1+\theta^2})\propto 1$, the uniform distribution. Because $\frac{1}{1+\theta^2}\in [0,1]$ for all $\theta\in(-\infty,\infty)$, our prior is that $\frac{1}{1+\theta^2}$ has the uniform distribution on $[0,1]$. This IS a proper prior.

Answer 2

Another counterexample:

Let $f(y|\theta) = \exp\{-(e^{-\theta} + \theta y)\} / y!$, where $\theta > 0$. The Fisher information for $\theta$ is $I(\theta)=e^{-\theta}$, so the Jeffreys prior is proportional to $e^{-\theta/2}$, i.e. an Exponential with rate 1/2.

[Yes, this is an extremely contrived example. $Y$ is just a Poisson with rate $e^{-\theta}$.]

Answer 3

The Fisher information matrix is related to the second order rate of change in the cross-entropy of a parametric distribution. When a distribution changes little as function of the parameter and even drops to zero, then you can have a parametric distribution with the Jeffreys prior being a proper distribution.

The examples in the other answers obtain such proper posterior distribution with a parametric distribution where the change in the distribution along the entire range of the parameter is only small. Another way to obtain a Jeffreys prior that is a proper distribution would be to use a reparameterisation of the Bernoulli distribution to obtain a paramter on an infinite range. For example $p = logit(\theta)$.

A more difficult case would probably be when we have a parametric distribution $f(x|\theta)$ such that the integral of the maximum frequency in each point $\int_{\forall x} max_{\theta} \lbrace f(x|\theta)\rbrace \,\text{d}x$ diverges.

That would place a minimum limit on the distribution such that each observation $x$ is at least likely for some parameter $\theta$. But, it is not that $max_{\theta} \lbrace f(x|\theta)\rbrace$ has a lower limit, for example for an exponential distribution $max_{\theta} \lbrace \theta e^{-\theta x} \rbrace = \frac{1}{e x}$ and this approaches zero for large $x$.