How to determine labeled balls for logistic regression

62 Views Asked by At

I am studying this paper about logistic regression. In section 4.2 (Randomly Generated Problems) on page 1534, they say "Features of positive (negative) examples are independent and identically distributed, drawn from a normal distribution $\mathcal{N}(\nu,1)$, where $\nu$ is in turn drawn from a uniform distribution on $[0,1]$ $([−1,0])$."

Question1: Why all entries of positive(negative) examples are drawn from the same distribution? More clearly, let $x\in \mathbb{R}^n$ be an example vector whose label is positive. Then all entries of $x$ are coming from a ball centered at $[\nu, \dots, \nu]^{\top}$ whose radius is roughly 1 because the variance is 1. Similarly, for an example vector whose label is negative all entries of $x$ are coming from a ball centered at $[-\nu, \dots, -\nu]^{\top}$ whose radius is roughly 1 because the variance is 1. Am I understand this correctly? The following is the picture of these balls when the standard deviation is 0.1. If we let the standard deviation be 1 they are overlapped. enter image description here

My thoughts: With the above set up we have two balls one in non-negative orthant and the other in non-positive orthant. Clearly, we can find a hyperplane that can separate them. This makes finding the separating hyperplane easy. We can have our two balls of data at any point of $\mathbb{R}^n$. To do that I have the following suggestion.

Question 2: Isn’t it better to sample each $i$-th entry of example vectors with positive label from $\mathcal{N}(\alpha_i,1)$ and $i$-th entry of example vectors with negative label from $\mathcal{N}(\beta_i,1)$? Then the center of example vectors with positive label would be $[\alpha_1,\dots,\alpha_n]^{\top}$ and the center of example vectors with negative label would be $[\beta_1,\dots,\beta_n]^{\top}$. Of course, $\alpha_i$'s and $\beta_i$'s are drawn from identically and independently from $\mathcal{N}(0,1)$. The following is one realization of what I explained. enter image description here

1

There are 1 best solutions below

5
On

Your picture about the balls is not correct. If $x$ is sampled from a normal distribution $N(0,1)$, then there is a substantial chance that $|x|$ will exceed 1. A more subtle point is that it sounds like $\nu$ is independently sampled for each data point.

Here is Python code to generate the examples like the describe:

import numpy as np
import matplotlib.pyplot as plt
N=200
Xpos=np.random.rand(N,2)+np.random.randn(N,2)
Xneg=-np.random.rand(N,2)+np.random.randn(N,2)
plt.scatter(*Xpos.T)
plt.scatter(*Xneg.T)`

and the result:

enter image description here