Calculating the plim of a posterior belief about a Bernoulli r.v. formed from observing an infinite sequence of other Bernoulli r.v.'s

Question

Suppose $Y$ is a Bernoulli random variable s.t. Prob$\{Y=1\} = \alpha\in(0,1)$. Further suppose I observe a sequence $\{X_t\}_{t=1}^\infty$ of i.i.d. Bernoulli random variables s.t. $\text{Prob} \{X_t=1|Y = y\} = \beta_y\in(0,1)$ $\forall y\in\{0,1\}$.

How would I calculate $\underset{t\to\infty}{\text{plim}}\ \text{Prob}\Big\{Y=1|\{X_{\tau}\}_{\tau=1}^t\ \Big\}?$ Thank you very much in advance!

P.S. If there are conditions (on $\alpha,\beta_0,\text{ and }\beta_1$) under which the above plim may not exist, I have a followup question. (But I suppose we'll cross that bridge when/if we get there!!) Thanks again!

Answer 1

Using Bayes' Theorem and the Law of Total Probability we have: $$\textrm{Prob}\left\{Y = 1 \mid \left\{X_\tau\right\}_{\tau=1}^t\right\} = \frac{\textrm{Prob}\left\{\left\{X_\tau\right\}_{\tau=1}^t \mid Y = 1\right\}\textrm{Prob}\left\{Y = 1\right\}}{\textrm{Prob}\left\{\left\{X_\tau\right\}_{\tau=1}^t \mid Y = 1\right\}\textrm{Prob}\left\{Y = 1\right\} + \textrm{Prob}\left\{\left\{X_\tau\right\}_{\tau=1}^t \mid Y = 0\right\}\textrm{Prob}\left\{Y = 0\right\}}.$$ Let $S_t := \sum_{\tau=1}^t X_\tau$. From your description we have $$\textrm{Prob}\left\{\left\{X_\tau\right\}_{\tau=1}^t \mid Y = y\right\} = \beta_y^{S_t} \left(1-\beta_y\right)^{t-S_t}.$$ Hence $$\textrm{Prob}\left\{Y = 1 \mid \left\{X_\tau\right\}_{\tau=1}^t\right\} = \frac{\beta_1^{S_t} \left(1-\beta_1\right)^{t-S_t}\alpha}{\beta_1^{S_t} \left(1-\beta_1\right)^{t-S_t}\alpha + \beta_0^{S_t} \left(1-\beta_0\right)^{t-S_t}(1-\alpha)}.$$ If $\beta_0 = \beta_1$ then this probability will equal $\alpha$ no matter what data we observe. Otherwise, let $$R_t = \frac{\beta_0^{S_t} \left(1-\beta_0\right)^{t-S_t}}{\beta_1^{S_t} \left(1-\beta_1\right)^{t-S_t}}.$$ Then $$\textrm{Prob}\left\{Y = 1 \mid \left\{X_\tau\right\}_{\tau=1}^t\right\} = \frac{\alpha}{\alpha + R_t(1-\alpha)}.$$ Now we need to work out what $R_t$ converges to. We first note that $S_t = t\bar{X}_t$. Given $Y = y$, we know that $\underset{t\to\infty}{\text{plim}}\ \bar{X}_t = \beta_y$.

Considering the case $Y = 1$, $$\underset{t\to\infty}{\text{plim}}\ R_t = \lim_{t\to\infty} \frac{\beta_0^{\beta_1t} \left(1-\beta_0\right)^{t\left(1-\beta_1\right)}}{\beta_1^{\beta_1t} \left(1-\beta_1\right)^{t\left(1-\beta_1\right)}} = \lim_{t\to\infty} \left[\frac{\beta_0^{\beta_1} \left(1-\beta_0\right)^{\left(1-\beta_1\right)}}{\beta_1^{\beta_1} \left(1-\beta_1\right)^{\left(1-\beta_1\right)}}\right]^t.$$ Using differentiation you can see that $x^{\beta_1}(1-x)^{1-\beta_1}$ is maximised when $x = \beta_1$. So $$\beta_0^{\beta_1} \left(1-\beta_0\right)^{\left(1-\beta_1\right)} < \beta_1^{\beta_1} \left(1-\beta_1\right)^{\left(1-\beta_1\right)}$$ and hence $$\underset{t\to\infty}{\text{plim}}\ R_t = \lim_{t\to\infty} \left[\frac{\beta_0^{\beta_1} \left(1-\beta_0\right)^{\left(1-\beta_1\right)}}{\beta_1^{\beta_1} \left(1-\beta_1\right)^{\left(1-\beta_1\right)}}\right]^t = 0,$$ and so $$\underset{t\to\infty}{\text{plim}}\ \textrm{Prob}\left\{Y = 1 \mid \left\{X_\tau\right\}_{\tau=1}^t\right\} = \underset{t\to\infty}{\text{plim}}\ \frac{\alpha}{\alpha + R_t(1-\alpha)} = 1.$$

Considering the case $Y=0$, we can use similar reasoning to see that $$\underset{t\to\infty}{\text{plim}}\ R_t = \infty$$ and so $$\underset{t\to\infty}{\text{plim}}\ \textrm{Prob}\left\{Y = 1 \mid \left\{X_\tau\right\}_{\tau=1}^t\right\} = \underset{t\to\infty}{\text{plim}}\ \frac{\alpha}{\alpha + R_t(1-\alpha)} = 0.$$

In conclusion,

• if $\beta_0 = \beta_1$ then $\underset{t\to\infty}{\text{plim}}\ \textrm{Prob}\left\{Y = 1 \mid \left\{X_\tau\right\}_{\tau=1}^t\right\} = \alpha$.
• Otherwise, if $Y = 1$ then $X_{\tau} \sim \textrm{Bernoulli}(\beta_1)$ and $\underset{t\to\infty}{\text{plim}}\ \textrm{Prob}\left\{Y = 1 \mid \left\{X_\tau\right\}_{\tau=1}^t\right\} = 1$.
• And if $Y = 0$ then $X_{\tau} \sim \textrm{Bernoulli}(\beta_0)$ and $\underset{t\to\infty}{\text{plim}}\ \textrm{Prob}\left\{Y = 1 \mid \left\{X_\tau\right\}_{\tau=1}^t\right\} = 0$.