Jeffreys prior for location-scale family of distributions - why is $\pi(\theta) \propto 1$ and $\pi(\sigma) \propto \frac{1}{\sigma}$?

167 Views Asked by At

I don't understand why the Jeffreys rule leads to $\pi(\theta) \propto 1$ and $\pi(\sigma) \propto \frac{1}{\sigma}$ in the below:


The Jeffreys prior is defined by $$ \pi(\theta) \propto I(\theta)^\frac{1}{2} $$ where $I(\theta)$ is the expected (Fisher) information.

Since $I(\theta) = n \times i(\theta)$ where $i(\theta)$ is the Fisher information in a sample of size $1$, the Jeffreys prior is $$ \pi(\theta) \propto i(\theta)^\frac{1}{2}. $$

Suppose $\theta \in \mathbb{R}$ and $f(x \mid \theta) = g(x-\theta)$ for some function $g$. Then $\theta$ is called a location parameter and Jeffreys rule leads to $\pi(\theta) \propto 1$ for $\theta \in \mathbb{R}$.

Suppose $\sigma >0$ and $f(x \mid \sigma) = \frac{1}{\sigma}g\left(\frac{x}{\sigma}\right)$ for some function $g$. Then $\sigma$ is called a scale parameter and Jeffreys rule leads to $\pi(\sigma) \propto \frac{1}{\sigma}$ for $\sigma >0$.

Now, if $f(x \mid \theta, \sigma) = \frac{1}{\sigma}g\left(\frac{x - \theta}{\sigma}\right)$ then we have a location-scale family of distributions. The approach of finding the Jeffreys prior for each component parameter separately and assuming prior independence of $\theta$ and $\sigma$ simply uses the product of the above two priors: $$\pi(\theta,\sigma) \propto 1 \times \frac{1}{\sigma} \quad \text{for } \theta \in \mathbb{R}, \sigma > 0.$$


EDIT [Attempted answer]

Case 1: $\theta \in \mathbb{R}$ and $f(x \mid \theta) = g(x-\theta)$ \begin{align} \frac{d^2}{d\theta^2}\log f(x \mid \theta) & = \frac{d^2}{d\theta^2}\log g(x-\theta)\\ &= \frac{d}{d\theta}\left(-\frac{g'(x-\theta)}{g(x-\theta)}\right)\\ &= \frac{g''(x-\theta)g(x-\theta) - g'(x-\theta)^2}{g(x-\theta)^2} \end{align} So \begin{align} i(\theta) &= E\left[-\frac{d^2}{d\theta^2}\log f(X \mid \theta)\right]\\ &= \int_{-\infty}^{\infty}\left(\frac{g'(x-\theta)^2 - g''(x-\theta)g(x-\theta)}{g(x-\theta)^2}\right)f(x \mid \theta) \,dx\\ &= \int_{-\infty}^{\infty}\left(\frac{g'(x-\theta)^2 - g''(x-\theta)g(x-\theta)}{g(x-\theta)^2}\right)g(x-\theta) \,dx\\ &= \int_{-\infty}^{\infty}\left(\frac{g'(x-\theta)^2}{g(x-\theta)} - g''(x-\theta)\right) \,dx \end{align} Changing variables to $u=x-\theta$, with $\frac{dx}{du} = 1$, we have $$ i(\theta) = \int_{-\infty}^{\infty}\left(\frac{g'(u)^2}{g(u)} - g''(u)\right) \,du $$ which is a constant that does not depend on $\theta$, i.e. $i(\theta) \propto 1$. Therefore, $\pi(\theta) \propto 1$.

Case 2: $\sigma >0$ and $f(x \mid \sigma) = \frac{1}{\sigma}g\left(\frac{x}{\sigma}\right)$ \begin{align} \frac{d^2}{d\sigma^2}\log f(x \mid \sigma) &= \frac{d^2}{d\sigma^2}\log\left(\frac{1}{\sigma}g\left(\frac{x}{\sigma}\right)\right)\\ &= \frac{d^2}{d\sigma^2}\left(-\log\sigma + \log g\left(\frac{x}{\sigma}\right)\right)\\ &= \frac{d}{d\sigma}\left(-\frac{1}{\sigma} - \frac{x g'\left(\frac{x}{\sigma}\right)}{\sigma^2 g\left(\frac{x}{\sigma}\right)}\right)\\ &= \frac{1}{\sigma^2} - x\frac{g''\left(\frac{x}{\sigma}\right)\left(-\frac{x}{\sigma^2}\right)\sigma^2 g\left(\frac{x}{\sigma}\right) - \left(2\sigma g\left(\frac{x}{\sigma}\right) + \sigma^2 g'\left(\frac{x}{\sigma}\right) \left(-\frac{x}{\sigma^2}\right)\right) g'\left(\frac{x}{\sigma}\right)}{\sigma^4 g\left(\frac{x}{\sigma}\right)^2}\\ &= \frac{1}{\sigma^2} + \frac{x^2 g''\left(\frac{x}{\sigma}\right) g\left(\frac{x}{\sigma}\right) + 2x\sigma g\left(\frac{x}{\sigma}\right)g'\left(\frac{x}{\sigma}\right) - x^2 g'\left(\frac{x}{\sigma}\right)^2}{\sigma^4 g\left(\frac{x}{\sigma}\right)^2} \end{align} So \begin{align} i(\sigma) &= E\left[-\frac{d^2}{d\sigma^2}\log f(X \mid \sigma)\right]\\ &= -\frac{1}{\sigma^2} - \int_{-\infty}^{\infty}\left(\frac{x^2 g''\left(\frac{x}{\sigma}\right) g\left(\frac{x}{\sigma}\right) + 2x\sigma g\left(\frac{x}{\sigma}\right)g'\left(\frac{x}{\sigma}\right) - x^2 g'\left(\frac{x}{\sigma}\right)^2}{\sigma^4 g\left(\frac{x}{\sigma}\right)^2}\right)f(x \mid \sigma) \,dx\\ &= -\frac{1}{\sigma^2} - \int_{-\infty}^{\infty}\left(\frac{x^2 g''\left(\frac{x}{\sigma}\right) g\left(\frac{x}{\sigma}\right) + 2x\sigma g\left(\frac{x}{\sigma}\right)g'\left(\frac{x}{\sigma}\right) - x^2 g'\left(\frac{x}{\sigma}\right)^2}{\sigma^4 g\left(\frac{x}{\sigma}\right)^2}\right)\frac{1}{\sigma}g\left(\frac{x}{\sigma}\right) \,dx \end{align} Changing variables to $u = \frac{x}{\sigma}$, with $\frac{dx}{du} = \sigma$, we have \begin{align} i(\sigma) &= -\frac{1}{\sigma^2} - \int_{-\infty}^{\infty}\left(\frac{\sigma^2 u^2 g''(u) g(u) + 2\sigma^2 u g(u)g'(u) - \sigma^2 u^2 g'(u)^2}{\sigma^4 g(u)^2}\right)g(u) \,du\\ &= -\frac{1}{\sigma^2} - \frac{1}{\sigma^2}\int_{-\infty}^{\infty}\left(u^2 g''(u) + 2u g'(u) - \frac{u^2 g'(u)^2}{g(u)}\right) \,du \end{align} where the integral is a constant that does not depend on $\sigma$, i.e. $i(\sigma) \propto \frac{1}{\sigma^2}$.

Therefore, $\pi(\sigma) \propto \frac{1}{\sigma}$.