Bayes Classifier with $2$ Normal Random Variables

708 Views Asked by At

For Class $1$ cases, $x$ will be an observation from a normal distribution having a mean of $0$ and a standard deviation of $1$, and for Class $2$ cases, $x$ will be an observation from a normal distribution having a mean of $0$ and a standard deviation of $2$. If Class $1$ cases are just as likely to be observed as Class $2$ cases, for what values of $x$ will a Bayes classifier predict Class $1$? Give the Bayes error rate for the Bayes classifier.

My understanding is that I need to set $\pi_1\cdot f_1(x) \gt \pi_2 \cdot f_2(x)$. Then

$$\begin{align*} \pi_1\cdot f_1(x) \gt \pi_2 \cdot f_2(x) & \iff \frac{1}{2}\cdot \frac{1}{\sqrt{2\pi}}e^{-\frac{x^2}{2}} \gt \frac{1}{2}\cdot \frac{1}{\sqrt{8\pi}}e^{-\frac{x^2}{8}}\\\\ & \iff \frac{\sqrt{8\pi}}{\sqrt{2\pi}} \gt \frac{e^{-\frac{x^2}{8}}}{e^{-\frac{x^2}{2}}}\\\\ & \iff 2 \gt e^{\frac{3x^2}{8}}\\\\ & \iff ln(2) \gt \frac{3x^2}{8} \\\\ & \iff \frac{8}{3}ln(2) \gt x^2 \end{align*}$$

So $$x\in(-1.36,1.36)$$

Would the error rate just be the probability of being outside of this interval if $X\sim N(0,1)$? In which case, I get from a normal table an error rate of $2\cdot0.0869 = 0.1738$

Edit:

Would the Bayes error rate be

$$\begin{align*} \frac{1}{2}(P(\vert N_1 \vert) \gt 1.36+P(\vert N_2 \vert) \lt 1.36) & =\frac{1}{2}(0.1738 + 0.503355) \\ &=.3387\\ \end{align*}$$

1

There are 1 best solutions below

5
On BEST ANSWER

Your calculation of the Bayes Classifier is correct; i.e. the Bayes Classifier is given to be

\begin{align*} C(x) = \begin{cases} 1 & \text{ if } |x| < \sqrt{\frac83 \log(2)}, \\ 0 & \text{else.} \end{cases} \end{align*}

The Bayes error is then defined to be the probability that the classifier is wrong. Letting $I(X)$ denote the true (unobserved) class of a sample $X$ this is the probability

\begin{align*} E & = \mathbf{P}\left[ C(X) \neq I(X)\right] \\ & = \mathbf{P}\left[ C(X) \neq I(X) \, | \, I(X) = 1 \right] \mathbf{P}[I(X) = 1] + \mathbf{P}\left[ C(X) \neq I(X) \, | \, I(X) = 2 \right] \mathbf{P}[I(X) = 2] \\ & = \frac12 \big( \mathbf{P}\left[ C(X) =2 \, | \, I(X) = 1 \right] + \mathbf{P}\left[ C(X) =1 \, | \, I(X) = 1 \right] \big) \end{align*} Letting $N_1 \sim N(0,1)$ and $N_2 \sim N(0,4)$ and $\alpha = \sqrt{\frac83 \log(2)} \sim1.36$, then the above probabilities are \begin{align*} \mathbf{P} \left[ C(X) = 2 \, | \, I(X) = 1\right] & = \mathbf{P}\left[ \, |N_1| > \alpha \right]\\ \mathbf{P} \left[ C(X) = 1 \, | \, I(X) = 2\right] & = \mathbf{P}\left[ \, |N_2| \leq \alpha \right] \end{align*} Denoting $\Phi$ for the CDF of the standard normal distribution we have \begin{align*} E & = \frac12 \big( \mathbf{P}\left[ \, |N_1| > \alpha \right] + \mathbf{P}\left[ \, |N_2| \leq \alpha \right] \big) \\ & = \frac12 \left( 1 - \left(\Phi(\alpha) - \Phi(-\alpha)\right) + \left(\Phi\left(\frac{\alpha}{2}\right) - \Phi\left(-\frac{\alpha}{2}\right) \right)\right) \\ & = \frac12 \mathbf P\left[ \, |N_1| \not\in \left( \frac{\alpha}{2}, \alpha\right) \right] \end{align*}

Note that the distinction to your solution, which is equivalent to $1 - (\Phi(\alpha) - \Phi(-\alpha) )$, is that you do not consider the case of misclassification as class $2$ when the variable is actually class $1$.