Suppose doctors are experimenting with two treatments without knowing which one is better. Call treatments $A$ and $B$. One is likely better, but we don’t know which one. A series of patients will each be given one of the treatments. We aim to find a strategy that ensures that as many as possible of the patients are given the better treatment — though we don’t know which one this is.
Given that for any patient, $\mathbb P( \text{ treatment A is success})=\alpha$ and $\mathbb P( \text{ treatment B is success})=\beta$, $0<\alpha, \beta<1$.
For any patient, doctors can choose with equal probability which treatment to go ahead with. Let $P_R$ be the overall probability of success for each patient. so $P_R=\frac{1}{2}(\alpha+\beta)$.
There is another strategy where for the first patient doctor chooses treatment $A$ and $B$ with equal probability at random; if the $n^{th}$ patient is given treatment $A$ and it is successful, they use the treatment $A$ for the $(n+1)^{th}$ patient too.
A similar rule is patient $n$ is given treatment $B$, and if it is successful, they use the treatment $B$ for the $(n+1)^{th}$ patient too. If not, switch to treatment $A$ for the $(n+1)^{th}$ patient.
So, the treatment system can be expressed with a Markov chain with $4$ states, right?
$\{A_s, A_f, B_s, B_f \}$
Could anyone tell me how if I have written the correct transition probabilities?
$\mathbb P( X_{n+1}=A_s \mid X_n=A_s)=\frac{1}{2}\alpha$
$\mathbb P( X_{n+1}=A_s \mid X_n=B_f)=\frac{1}{4}\alpha(1-\beta)$
$\mathbb P( X_{n+1}=A_s \mid X_n=B_s)=0$
$\mathbb P( X_{n+1}=B_s \mid X_n=B_s)=\frac{1}{2}\beta$
$\mathbb P( X_{n+1}=B_s \mid X_n=A_f)=\frac{1}{4}(1-\alpha)\beta$
$\mathbb P( X_{n+1}=B_s \mid X_n=A_s)=0$
Where $A_s$ means that the patient was given treatment $A$ and was successful. $f$ indicates failure.
Define $P_T$ to be the long-run probability of success in this strategy.
Could anyone tell me how to find the equilibrium distribution $\pi$ for the process? Find $P_T$, the long-run probability of success for each patient in this strategy.
Thank you for your help.
Inaccuracies in the transitions
I don't think your transition probabilities are correct.
For instance, condition on $X_n = A_s$, $X_{n+1} \in \{A_s, A_f\}$, but you haven't defined a probability for $\mathbf P[X_{n+1} = A_f | X_n = A_s]$.
Further, still conditioning on $X_n = A_s$ the respective probabilities are simply the probability $A$ succeeds/fails, so:
$$\begin{align*} \mathbf P[X_{n+1} = A_s | X_n = A_s] & = \alpha,\\ \mathbf P[X_{n+1} = A_f | X_n = A_s] & = 1-\alpha. \end{align*}$$ So we don't need the factor $\frac12$ you have in your first formula.
Complete Transition Matrix
Let $P$ denote the transition matrix, and I'll assume the order of the four states that you provided $\{A_s, A_f,B_s, B_f\}$, then thinking through each scenario, as above, you can show:
$$P = \left( \begin{matrix} \alpha & 1-\alpha & 0 & 0 \\ 0 & 0 & \beta & 1-\beta \\ 0 & 0 & \beta & 1-\beta\\ \alpha & 1-\alpha & 0 & 0 \end{matrix} \right)$$
Calculating the stationary distribution This is an irreducible, aperiodic, Markov chain on a finite state space, so it is guaranteed to have a unique stationary distribution, $\pi$ (e.g. Theorem 4.9 in Levin and Peres).
Further, by definition of a stationary distribution, it satisfies:
$$\pi P = \pi.$$
Applying this to the matrix above, we have the system of equations
$$ \begin{align*} \alpha(\pi_1 + \pi_4) = \pi_1 \\ (1-\alpha)(\pi_1 + \pi_4) = \pi_2 \\ \beta(\pi_2 + \pi_3) = \pi_3 \\ (1-\beta)(\pi_2 + \pi_3) = \pi_4 \end{align*} $$
Combining the first two equations, we find that $\pi_2 = \pi_4$, which we'll denote $p$. Then respectively substituting $p$ for $\pi_4$ for $\pi_2$ in the second and fourth equations we find:
$$ \begin{align*} \pi_1 = \frac{\alpha}{1-\alpha}p \\ \pi_3 = \frac{\beta}{1-\beta}p \end{align*} $$
Since $\pi_1 + \pi_2 + \pi_3 + \pi_4 =1$ we have:
$$1 =\frac{\alpha}{1-\alpha}p + p + \frac{\beta}{1-\beta}p + p,$$
Equivalently:
$$p = \frac{1}{2 + \frac{\alpha}{1-\alpha} + \frac{\beta}{1-\beta}}.$$
So we have the stationary distribution:
$$ \pi = \frac{1}{2 + \frac{\alpha}{1-\alpha} + \frac{\beta}{1-\beta}} \left(\begin{matrix} \frac{\alpha}{1-\alpha} \\ 1 \\ \frac{\beta}{1-\beta} \\ 1 \end{matrix}\right) $$
Success Probability
Having run the Markov chain long enough, (approximately) any patient can expect to be in each state according to the stationary distribution $\pi$. Successful patients are those in states $A_s,\, B_s$, which correspond to $\pi_1,\pi_3$.
So the probability of success in this strategy is
$$ \pi_1 + \pi_3 = \frac{\frac{\alpha}{1-\alpha} + \frac{\beta}{1-\beta}}{2 + \frac{\alpha}{1-\alpha} + \frac{\beta}{1-\beta}}=\frac{\alpha+\beta-2\alpha\beta}{2-\alpha-\beta}$$