Hard vs Soft accuracy: Bounds in specific probability problem

204 Views Asked by At

Let $\mathbf{X}=(X_1,\cdots,X_m)$ a random vector which takes values in $(0,1)^m$ such that $\sum_{i=1}^mX_i=1$ (a.s.) and let $Y$ a random variable which takes values in $\{1,\cdots,m\}$. Let $s=\mathbb{E}[X_Y]$ and $h=\mathbb{P}\left(Y\in\mathop{\arg\max}_i X_i\right)$, I need bounds that relate these magnitudes in the general case.

MOTIVATION: In Machine Learning context, $s$ and $h$ are the soft and hard accuracy. Usually we take the hard one, so we assume that $h>s$. I want to prove it.

The tightest bound that I found is: $$s=\int_0^1\mathbf{P}(X_Y>t)dt=\int_0^\frac{1}{2}\mathbf{P}(X_Y>t)dt+\int_\frac{1}{2}^1\mathbf{P}(X_Y>t)dt\leq\frac{1}{2}+\frac{1}{2}\mathbf{P}(X_Y>1/2)\leq \frac{1+h}{2}$$ where I use that if some value is greater than $1/2$, it is the maximum.

Q1) Is there another upper bound of $s$ tighter than this? (suppose $h$ and $s$ are values close to $1$).

Q2) I would like $s\leq h$. Can you think of a counterexample?

Q3) Can you think extra assumptions to ensure $s\leq h$?

Thanks

1

There are 1 best solutions below

0
On

I have a possible answer to Q3. Let $s>1/2$ and $\frac{1}{2}<M\leq s$ where $M=\mathbb{M}(X_Y)$ is the median of $X_Y$. Then it is easy to check the following inequality: $$\frac{1}{2}\leq \mathbb{P}(X_Y\geq M)\leq \mathbb{P}(X_Y> 1/2)\leq h$$
where I use that if some value is greater than 1/2, it is the maximum. Then we can bound, $$s=\int_0^{1/2}\mathbb{P}(X_Y> t)dt+\int_{1/2}^M\mathbb{P}(X_Y> t)dt+\int_M^{1}\mathbb{P}(X_Y> t)dt$$ $$\leq\frac{1}{2}+\left(M-\frac{1}{2}\right)\mathbb{P}(X_Y> 1/2)+(1-M)\mathbb{P}(X_Y>M)$$ $$\leq\frac{1}{2}+\left(M-\frac{1}{2}\right)h+\frac{1-M}{2}=1-\frac{h}{2}+M\left(h-\frac{1}{2}\right)$$ $$\leq1-\frac{h}{2}+s\left(h-\frac{1}{2}\right)$$ where I use $h\geq1/2$. Then we can solve: $$s\leq\frac{2-h}{3-2h}=h-\mathcal{O}((h-1)^2)$$

Any possible improvement?