Gaussian Class-Conditional distribution
(a) Consider a binary classifier (i.e. y ∈ {0, 1}). Suppose the class conditional distribution, $p(x^{(n)}|y^{(n)} = j)$ is a Gaussian with mean $µ_{j}$ and covariance matrix $Σ_{m}$, for j = 1, 2. Let δ be the decision boundary between two classes, i.e.
δ(x) = $\log\frac{p(y = 1|x)}{p(y = 2|x)}$
Further suppose that the prior over y is uniform. Write the Bayes classifier as y = f(x) = sign(δ(X)) and simplify δ as much as possible. What is the geometric shape of the decision boundary?
(b) Repeat (a) but assume that the two Gaussians have identical covariance matrices. What is the geometric shape of the decision boundary?
(c) Repeat (a) but assume now that the two Gaussians have covariance matrix which is equal to the identity matrix ($Σ_{1} = Σ_{2} = I$). What is the geometric shape of the decision boundary?
Note: Recall that the p-dimensional Gaussian distribution has the form
N(x; µ, Σ) = $\frac{1}{(2π)^{p/2} |Σ|^{1/2}}$ $e^{-\frac{1}{2}(x − µ)^{T} Σ^{-1}(x − µ)}$
EDIT: made edit to the formulas because I forgot the log normalizing constants
The decision boundary is when one of the two possible Gaussians is more likely than the other. You can see this in the formula $\log\frac{P[y=1|x]}{P[y=2|x]}$.
The log will be negative if $P[y=1|x]<P[y=2|x]$ and nonnegative when $P[y=1|x] \geq P[y=2|x]$.
The decision boundary is when they are equal, giving $\delta(x) = 0$.
Taking the log of the ratio of the two densities gives us:
\begin{equation} \begin{split} & \log\frac{P[y=1|x]}{P[y=2|x]} = 0 \\ \Leftrightarrow & (x-\mu_1)\Sigma^{-1}_1(x-\mu_1) - (x-\mu_2)\Sigma^{-1}_2(x-\mu_2) + c = 0 \\ \Leftrightarrow & \sum_{i=1}^p(x-\mu_{i_1})^2\lambda_{ii_1} + \sum_{i\neq j}(x-\mu_{i_1})(x-\mu_{j_1})\lambda_{ij_1} - \sum_{i=1}^p(x-\mu_{i_2})^2\lambda_{ii_2} - \sum_{i\neq j}(x-\mu_{i_2})(x-\mu_{j_2})\lambda_{ij_2} + c = 0 \end{split} \end{equation}
Where $\Lambda = \Sigma^{-1}$ and $\lambda_{ij}$ is the $i,j$ element of $\Lambda$, and $c$ is a constant representing the log of the normalizing constants: $c = \frac{1}{2}\log|\Sigma_1| - \frac{1}{2}\log|\Sigma_2|$.
Note that this is the difference of two quadratic polynomials in $x$, which is also a quadratic polynomial in $x$. So the shape of the decision boundary is the set of solutions to this polynomial.
You should be able to use that formula to solve the last two questions by adjusting the parameters.