Machine learning with Reject Option

Question

Can any one have ideas of machine learning with Reject Option (ambiguous points)?

Are there any software package for some of these algorithms? ie, how to choose the reject threshold

Answer 1

Assuming you mean the option for the machine to output "unsure", it depends on whether you have labels or not.

If you have "ambiguous" as a label in your training set (or you can generate samples that are ambiguous, e.g. via GANs), then you can simply do standard ML classification with "ambiguous" as a label.

If you cannot do this, then I think there are two main options:

1. Use a probabilistic model for anomaly detection. E.g. see here. Essentially, if the outlier detector detects an outlier, declare the point to be ambiguous. (This classifier is separate from your regular learner). For instance, you could fit Gaussian mixtures to each class dataset; given a new point, if the probability of any model $P(x|C_i)$ is too small for every class $i$, then you can call $x$ an outlier and output "ambiguous".

2. Quantify the uncertainty in your learner itself, directly. For instance, some people doing $K$-way classification get a (softmaxed) output $P(y|x) = (p_1,\ldots,p_K)$, then if $\max_j p_j < T$ for some threshold $T$ (say 0.5), we call $x$ an outlier. The main problem with this is the calibration of the network output as probabilities is not very good. A better approach is to use a Bayesian neural network (BNN); check out the Edward library, for instance. For classification, we can get a well-calibrated estimate of $P(y|x)$; for regression, one can get a confidence interval and/or variance for a given prediction (i.e. $\mathbb{V}[f(x)]$).