Let $X$ be a random variable with a probability density $f(\cdot;\theta)$ where $\theta \in \Theta \subset \mathbb R$ is an unknown parameter. Suppose that we have a prior density $\pi(\theta)$, with associated random variable for the parameter $\tilde\Theta$. Suppose that we have a random sample $X_1, \dots, X_n$ from $X$.
The interest is in the hypothesis testing problem $$ H_0: \theta \in \Theta_0 \quad\text{vs.}\quad H_1: \theta \in \Theta_1,$$ where $\Theta_1 = \Theta \setminus \Theta_0$.
Now, according to my course notes, we should proceed as follows. We determine the ratio of the posterior probabilities
$$ \frac{P(\tilde\Theta \in \Theta_0 \mid X_1,\dots,X_n)}{P(\tilde\Theta \in \Theta_1 \mid X_1,\dots,X_n)} = \frac{P(\tilde\Theta \in \Theta_0) \int_{\Theta_0} \prod_{i=1}^n f(X_i;\theta) \pi(\theta) \,d\theta}{P(\tilde\Theta \in \Theta_1) \int_{\Theta_1} \prod_{i=1}^n f(X_i;\theta) \pi(\theta) \,d\theta}.$$
So we have $$ \frac{P(\tilde\Theta \in \Theta_0 \mid X_1,\dots,X_n)}{P(\tilde\Theta \in \Theta_1 \mid X_1,\dots,X_n)} = \frac{ \int_{\Theta_0} \prod_{i=1}^n f(X_i;\theta) \pi(\theta) \,d\theta}{ \int_{\Theta_1} \prod_{i=1}^n f(X_i;\theta) \pi(\theta) \,d\theta} \times \frac{\int_{\Theta_0} \pi(\theta)}{1 - \int_{\Theta_0} \pi(\theta)}.$$
However, we also have that the posterior density is given by $$f_{\tilde \Theta \mid X_1,\dots,X_n}(\theta\mid x_1,\dots,x_n) = \frac{\prod_{i=1}^n f(X_i;\theta) \pi(\theta)}{\int_{\Theta}\prod_{i=1}^n f(X_i;\theta) \pi(\theta)\,d\theta}.$$
So it seems to me that the factor $\frac{ \int_{\Theta_0} \prod_{i=1}^n f(X_i;\theta) \pi(\theta) \,d\theta}{ \int_{\Theta_1} \prod_{i=1}^n f(X_i;\theta) \pi(\theta) \,d\theta}$ is already the ratio of the posterior probabilities (since it is just the posterior distribution integrated over $\Theta_0$ and $\Theta_1$). Where is the mistake in my reasoning?
You’re right, and the course notes are wrong.
You can perhaps see this most clearly if you take both $f$ and $\pi$ constant. Then they both drop out of these equations, and the course notes are left with
$$ \frac{P(\tilde\Theta \in \Theta_0 \mid X_1,\dots,X_n)}{P(\tilde\Theta \in \Theta_1 \mid X_1,\dots,X_n)} = \frac{P(\tilde\Theta \in \Theta_0) \int_{\Theta_0} \mathrm d\theta}{P(\tilde\Theta \in \Theta_1) \int_{\Theta_1} \,\mathrm d\theta}. $$
But now the left-hand side is
$$ \frac{P(\tilde\Theta \in \Theta_0)}{P(\tilde\Theta \in \Theta_1)}\;, $$
whereas the right-hand side is
$$ \frac{P(\tilde\Theta \in \Theta_0)^2}{P(\tilde\Theta \in \Theta_1)^2}\;. $$