let $X \sim Bin(10,p)$
We would like to test the following hypotheses:
$H_{0}: p=0.5 $ vs $H_{1} : p=0.3$
The Statistic $T(x)$ is defined as $T(x)=x$ and the critical (rejection) region is $R=\{x:T(x)\le4\}$
I want to calculate the conditional probability of the null hypothesis given $T(x)\in R $
That is : $P(H_{0}\mid T(x)\in R)$
My first solution is: $P(H_{0}\mid T(x)\in R)=\frac{\alpha}{\alpha+(1-\beta)}$
My second solution (according to Bayesian hypothesis):
$P(H_{0}\mid T(x)\in R) = P(H_{0})*L(H_{0}\mid T(x)\in R)$ , which is equal to :
$P(H_{0}\mid T(x)\in R) = P(H_{0})*P(T(x)\in R\mid H_{0})$
$P(T(x)\in R\mid H_{0})= P_{H_{0}}(X\le4)=0.37695=\alpha$
while $P(H_{0})$ equal to :
$ P(H_{0})= P(H_{0}\mid T(x)\notin R) + P(H_{0}\mid T(x)\in R)=\frac{1-\alpha}{\beta} +\frac{\alpha}{1-\beta} $
Thus: $P(H_{0}\mid T(x)\in R)=(\frac{1-\alpha}{\beta} +\frac{\alpha}{1-\beta})*\alpha$
Which of the above solutions is correct ?
None of this makes any sense. The frequentist would say $\Pr[H_0]$ is not well-defined because $\Pr[p = 0.5]$ is not a statement about a random variable; and the Bayesian would protest that you have not specified a suitable prior distribution for $p$, and the computation cannot be performed until you do.
You also do not explain how you reason $$\Pr[H_0 \mid T(x) \in R] = \frac{\alpha}{1-\beta}.$$ Again, the frequentist will tell you this is nonsensical because $p$ is a parameter: it is fixed, and does not change from experiment to experiment. That it is unknown to us does not make it a random variable. The Bayesian would tell you that $$\Pr[H_0 \mid T(x) \in R] = \Pr[p = 0.5 \mid X \le 4] = \frac{\Pr[X \le 4 \mid p = 0.5]\Pr[p = 0.5]}{\Pr[X \le 4]}$$ and could reasonably interpret your hypothesis to imply that the prior on $p$ is a location-scale transformed Bernoulli with hyperparameter $\theta$; e.g. $$\Pr[p = 0.5] = 1-\theta, \quad \Pr[p = 0.3] = \theta$$ hence $$\begin{align} \Pr[X \le 4] &= \Pr[X \le 4 \mid p = 0.5] \Pr[p = 0.5] + \Pr[X \le 4 \mid p = 0.3] \Pr[p = 0.3] \\ &= \alpha (1 - \theta) + (1 - \beta) \theta \end{align}$$ and $$\Pr[H_0 \mid T(x) \in R] = \frac{\alpha(1-\theta)}{\alpha(1-\theta) + (1-\beta)\theta}$$ is the posterior for $H_0$ given rejection. Your computation is incorrect for the reason that you did not apply the law of total probability correctly: $$\Pr[H_0] = \Pr[H_0 \mid T(x) \in R]\color{red}{\Pr[T(x) \in R]} + \Pr[H_0 \mid T(x) \not\in R]\color{red}{\Pr[T(x) \not\in R]}$$ and under a Bayesian interpretation, you cannot calculate these marginal probabilities in red without specifying a prior distribution on $p$.