Calculating posterior with Improper uniform prior

Question

There is a random variable $\theta \in \mathbb{R}$ and assume an improper uniform prior for $\theta.$ Let $\epsilon \sim N(0,1),$ independent of $\theta.$ Then you're given the noisy signal $s = \theta + \epsilon.$ We want to find $Pr(\theta < \theta^* \mid s)$ where $\theta^*$ is some known threshold value.

My old notes calculates this to be $Pr(\epsilon > s - \theta^*) = 1 - \Phi(s-\theta^*),$ but I think we should be getting $Pr(\theta < \theta^* \mid s) = Pr(\epsilon > s - \theta^* \mid s)$ instead, conditioning on $s,$ but I'm not sure how this is evaluated using the improper prior of $\theta.$

Answer 1

I think your notation is causing your confusion, so I will rewrite some of it.

$\Theta$ is your improper prior and $\epsilon$ is your standard normal noise. Then $S = \Theta + \epsilon$ is the signal. You want to compute $$\Pr[\Theta < \theta^* \mid S = s].$$ In other words, given that you observed the signal $S = s$, you want the probability that $\Theta$ is less than some known $\theta^*$. Then this is $$\Pr[\Theta < \theta^* \mid S = s] = \Pr[S - \epsilon < \theta^* \mid S = s] = \Pr[s - \epsilon < \theta^*] = \Pr[\epsilon > s - \theta^*].$$ The first equality is simply the application of $S = \Theta + \epsilon$ by definition. The second follows by applying the conditional statement $S = s$ to the criterion $S - \epsilon < \theta^*$. After we apply it, because $\epsilon$ is independent noise and $\theta^*$ is a constant, it is no longer required to write the conditional $S = s$. The third equality is simply algebraic rearrangement, now noting that $s$ and $\theta^*$ are fixed constants and $\epsilon$ is the only random variable in the probability statement.

Answer 2

I've figured it out. By definition, the posterior distribution is $$f_{\Theta \mid S} (\theta \mid s) = \frac{ f_{\Theta} (\theta) f_{S \mid \Theta} (s \mid \theta) }{ \int_{\mathbb{R}} f_{\Theta} (\theta) f_{S \mid \Theta} (s \mid \theta) d \theta }$$

The conditional distribution density $f_{S \mid \Theta} (s \mid \theta) = \phi(s - \theta)$ since $S = \Theta + \epsilon$ and $\epsilon \sim N(0,1).$ Using the uniform prior $f_{\Theta} (\theta) = 1,$ we have $$f_{\Theta \mid S} (\theta \mid s) = \frac{f_{\Theta} (\theta) f_{S \mid \Theta} (s \mid \theta) }{\int_{\mathbb{R}} f_{\Theta} (\theta) f_{S \mid \Theta} (s \mid \theta) d \theta} = \frac{1 \cdot f_{S \mid \Theta} (s \mid \theta) }{\int_{\mathbb{R}} 1 \cdot f_{S \mid \Theta} (s \mid \theta) d \theta} = \frac{f_{S \mid \Theta} (s \mid \theta) }{1} = f_{S \mid \Theta} (s \mid \theta) = \phi(s - \theta),$$ so $\Theta \mid S \sim N(s, 1).$

So thus $Pr(\Theta < \theta^* \mid s) = \int_{-\infty}^{\theta^*} f_{\Theta \mid S} (\theta \mid s) d \theta= \int_{-\infty}^{\theta^*} \phi(s - \theta) d \theta = 1 - \Phi(s - \theta^*).$

It should be noted that the result is dependent on the prior being improper uniform on $\mathbb{R},$ as in general $f_{X \mid Y} (x \mid y) \neq f_{Y \mid X} (y \mid x).$

For example, if instead $\Theta \in [a,b]$ and our prior is $U[a,b],$ we would get the posterior equals (after cancelling $f_{\Theta} (\theta) = \frac{1}{b-a}$ from numerator and denominator) $$f_{\Theta \mid S} (\theta \mid s) = \frac{ f_{S \mid \Theta} (s \mid \theta) }{ \int_{a}^{b} f_{S \mid \Theta} (s \mid \theta) d \theta} = \frac{\phi(s-\theta)}{\int_{a}^{b} \phi(s-\theta) d\theta} = \frac{\phi(s-\theta)}{\Phi(s-a) - \Phi(s-b)} \neq \phi(s-\theta) = f_{S \mid \Theta} (s \mid \theta)$$ Though as we let this approach the improper uniform prior, $a \to -\infty, b \to \infty,$ this does approach $\phi(s-\theta) / (1-0) = \phi(s-\theta)$ as expected.