In Chapter 2 of [Duda,Hart,Stork]'s Pattern Classification book, it was discussed that full error of Bayes' decision rule in 2 category case is $P(error)=\int P(error\mid x) P(x) dx$. In part (a) of question 1 of the same chapter, the author asked us to replace $P(error \mid x)=2P(\omega_1 \mid x)P(\omega_2 \mid x)$ to get an upper bound on the full error.
Can somebody help me to see how the hint / replacement leads to the upper bound? I have tried to search around and suspect that it might have to do with the Chernoff bound but I can't establish the link.
Thank in in advance.
Unfortunately as you have stated the question above, it is difficult to follow without referring directly to the original reference.
In the above you miss a key fact which is that in the two category case (that a variable is $\omega_1$ or $\omega_2$), if we use the (Bayes) decision rule
$$ R(x) = \begin{cases} \omega_1 & \text{if $\mathbf P(\omega_1 \, | \, x) > \mathbf P(\omega_2 \, | \, x)$,} \\ \omega_2 & \text{else.} \end{cases}$$
Then the probability of an error (i.e. $\omega_1$ when the decision rule $R(x) = \omega_2$, and visa versa) conditioned on $x$ is given by
\begin{align*} \mathbf{P}(\text{error}\, | \, x) & = \min \big(\mathbf P(\omega_1 \, | \,x), \mathbf P (\omega_2 \, | \, x) \big) \\ & = \min \big(\mathbf P(\omega_1 \, | \,x), 1 - \mathbf P(\omega_1 \, | \,x) \big) \end{align*}
This formula is essential to solving the problem, since now if we denote $p = \mathbf P(\omega_1 \, | \,x) \in [0,1]$ then the above formula says $\mathbf P(\text{error} | x) = \min(p,1-p)$.
If we denote $f_\alpha(p) = \alpha p(1-p)$, the question (parts a-d, not just part a) is asking you to prove that for all $p \in [0,1]$
$$ f_1(p) \leq \min(p,1-p) \leq f_2(p),$$
so that $f_1$ is a lower bound and $f_2$ an upper bound on the error. Further that for any $\alpha \in (0,2)$, $f_\alpha$ is not a bound. That is, there exist points $q,r \in [0,1]$ such that \begin{align*} f_\alpha(q) & < \min\big(q,(1-q)\big) \\ f_\alpha(r) & > \min\big(r,(1-r)\big). \end{align*}
The graph below demonstrates this. Hopefully this is enough of a hint for you to now solve the problem.