Illustrate the invariance property of a noninformative prior

495 Views Asked by At

Consider $n$ i.i.d observations from a normal distribution with unknown mean, $\mu$, and unknown variance $\sigma^2$, ie, $y_i \sim i.i.d \ N(\mu, \sigma^2)$ for $i = 1, 2, \cdots, n$.

Let $\boldsymbol{\theta} = (\mu, \sigma)'$ be the parameterisation for the two unknown parameters. Using Jeffrey's Prior and assuming a priori independence between $\mu$ and $\sigma$, we can show that a noninformative prior for $\boldsymbol{\theta}$ can be written as $p(\mu, \sigma) \propto \frac{1}{\sigma}$. The posterior is thus given by: \begin{align*} p(\mu, \sigma|\mathbf{y}) & \propto L(\mu, \sigma|\mathbf{y})p(\mu,\sigma) \ \ \text{where} \ \ L \ \ \text{denotes the likelihood function}. \\ & \propto \frac{1}{\sigma^n}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\mu)^2\right] \times \frac{1}{\sigma} \\ & = \frac{1}{\sigma^{n+1}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\mu)^2\right] \end{align*}

I want to show that this prior is invariant to the parameterisation where we define $\boldsymbol{\eta} = (\mu, \psi)'$ where $\psi = \ln(\sigma)$. The following is my working, please advise if my arguments and reasoning are correct.

Given $\boldsymbol{\theta} = (\mu, \sigma)$ and $\boldsymbol{\eta} = (\mu, \psi)$ where $\psi = \ln(\sigma) \implies \exp(\psi) = \sigma$, to show that invariance holds, we need to show that $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\eta}|\mathbf{y})p(\boldsymbol{\eta})$ and $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\theta}|\mathbf{y})p(\boldsymbol{\theta})\displaystyle{\left|\frac{\partial \boldsymbol{\theta}}{\partial \boldsymbol{\eta}'}\right|}$ provide equivalent expressions. We begin by finding an expression for $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\eta}|\mathbf{y})p(\boldsymbol{\eta})$.

Since, $$p(\mathbf{y}|\boldsymbol{\theta}) = L(\boldsymbol{\theta}|\mathbf{y}) = \left(2\pi\right)^{-\frac{n}{2}}\left(\sigma^2\right)^{-\frac{n}{2}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n (y_i - \mu)^2\right]$$ Then, $$L(\boldsymbol{\eta}|\mathbf{y}) = \left(2\pi\right)^{-\frac{n}{2}}\left(\exp(2\psi)\right)^{-\frac{n}{2}}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n (y_i - \mu)^2\right]$$ The log-likelihood is thus, $$l(\boldsymbol{\eta}|\mathbf{y}) = -\frac{n}{2}\ln(2\pi) - \frac{n}{2}\left(2\psi\right) - \frac{1}{2\exp(2\psi)}\sum_{i=1}^n (y_i - \mu)^2$$ Assuming a priori independence between $\mu$ and $\psi$ implies $p(\mu, \psi) = p(\mu)p(\psi)$, and applying Jeffreys' Prior on each parameter yields, \begin{align*}p(\mu) & \propto \left|-E\left[\frac{\partial^2 l}{\partial \mu^2}\right]\right|^{\frac{1}{2}} \\ p(\psi) & \propto \left|-E\left[\frac{\partial^2 l}{\partial \psi^2}\right]\right|^{\frac{1}{2}}\end{align*}

Since $\frac{\partial^2 l}{\partial \mu^2} = -\frac{n}{\exp(2\psi)}$ and $\frac{\partial^2 l}{\partial \psi^2} = -\frac{2}{\exp(2\psi)}\sum_{i=1}^n (y_i-\mu)^2$, we have: \begin{align*} p(\mu) \propto \left|-E\left[-\frac{n}{\exp(2\psi)}\right]\right|^{\frac{1}{2}} = \left(\frac{n}{\exp(2\psi)}\right)^{\frac{1}{2}} \propto c \ \ \ \text{where} \ c \ \text{is a constant} \end{align*} $$p(\psi) \propto \left|-E\left[-\frac{2}{\exp(2\psi)}\sum_{i=1}^n (y_i-\mu)^2\right]\right|^{\frac{1}{2}} =\left(\frac{2}{\exp(2\psi)}E\left[\sum_{i=1}^n (y_i-\mu)^2\right]\right)^{\frac{1}{2}} = \left|\frac{2n\exp(2\psi)}{\exp(2\psi)} \right|^{\frac{1}{2}} \propto c$$

Hence $p(\boldsymbol{\eta}) \propto c$, and $p(\boldsymbol{\eta}|\mathbf{y}) \propto \frac{1}{\exp(\psi n)}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n (y_i - \mu)^2\right]$. Next we find an expression for $p(\boldsymbol{\eta}|\mathbf{y}) \propto L(\boldsymbol{\theta}|\mathbf{y})p(\boldsymbol{\theta})\displaystyle{\left|\frac{\partial \boldsymbol{\theta}}{\partial \boldsymbol{\eta}'}\right|}$, assuming we use the correct, non-informative prior for $\boldsymbol{\theta}$, ie, $p(\boldsymbol{\theta}) \propto \frac{1}{\sigma}$. Computing the Jacobian is as follows: \begin{align*} \displaystyle{\left|\frac{\partial \boldsymbol{\theta}}{\partial \boldsymbol{\eta}'}\right|} = \left|\left[\begin{matrix} \mu \\ \sigma \end{matrix}\right]\left[\begin{matrix} \frac{\partial}{\partial \mu} & \frac{\partial}{\partial \psi} \end{matrix}\right] \right| = \left|\left[\begin{matrix} \frac{\partial \mu}{\partial \mu} & \frac{\partial \mu}{\partial \psi} \\ \frac{\partial \sigma}{\partial \mu} & \frac{\partial \sigma}{\partial \psi} \end{matrix}\right]\right| = \left|\left[\begin{matrix} 1 & 0 \\ 0 & \frac{\partial\exp(\psi)}{\partial \psi} \end{matrix}\right]\right| = \left|\left[\begin{matrix} 1 & 0 \\ 0 & \exp(\psi) \end{matrix}\right]\right| = \exp(\psi) \end{align*}

Thus we have, \begin{align*} p(\boldsymbol{\eta}|\mathbf{y}) & \propto \frac{1}{\sigma^{n+1}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\mu)^2\right]\exp(\psi) \\ &= \frac{1}{\exp(\psi n + \psi)}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n(y_i-\mu)^2\right]\exp(\psi) \\ &= \frac{1}{\exp(\psi n)}\exp\left[-\frac{1}{2\exp(2\psi)}\sum_{i=1}^n(y_i-\mu)^2\right] \end{align*}

Which is exactly the expression we obtained, hence the invariance property holds if $p(\mu, \sigma) \propto \frac{1}{\sigma}$.

1

There are 1 best solutions below

1
On BEST ANSWER

To "Illustrate the invariance property of a noninformative prior" (assuming by noninformative you mean Jeffreys) just work with the general case of Jeffreys prior being the square root of the determinant of the fisher information and take the jacobian inside the square root. I would write it out, but wikipedia has already done it

http://en.wikipedia.org/wiki/Jeffreys_prior

that should be enough to show you that it is invariant under reparameterization.