Properties of the solution of a linear system with random equations

48 Views Asked by At

$x_i$ is drawn from $\mathrm{unif}(a,b)$, $y_i$ is drawn from $\mathrm{unif}(c,d)$.

$x_i$ are independent from each other. $y_i$ are independent from each other. $x_i$ are independent of $y_i$.

$i$ goes from $1$ to $n$.

I have the linear system

$$y_i=\alpha \cdot x_i+\beta$$

I solve the linear system for $\alpha$ and $\beta$ with least squares; what properties do the estimates of $\alpha$ and $\beta$ have?

1

There are 1 best solutions below

1
On

I'll asume that you're drawing points uniformly from the rectangle $[a,b]\times[c,d]$ (which means that all the variables you mention are independent). We can't expect any kind of line-like structure to appear in the points you draw. $\alpha$ can be anything at all, and will be extremely sensitive to the actual points you have drawn. Since there is no reason for the resulting line to be increasing instead of decreasing, or vice-versa, the mean of $\alpha$ is $0$ (that is, if the mean exists).

$\beta$ is the point at which the line crosses the $y$ axis. If the interval $[a,b]$ is far away from the origin, $\beta$ will be pretty much random, but if the interval is near the origin, $\beta$ will get more concentrated around its mean, which is $(c+d)/2$ (by the same symmetry argument).

Anyway, as random as $\alpha$ and $\beta$ can be, they will be highly correlated if the rectangle is far away from the origin (because the line given by the least-squares model will be pretty much random, but it is guaranteed to intersect the rectangle, and if you have lots and lots of points, it will very probably pass very near the center of the rectangle). For example: if $[a,b]\times [c,d]=[-1-\varepsilon,-1+\varepsilon]\times [-\varepsilon,\varepsilon]$, with $\varepsilon$ very small, then $\alpha$ and $\beta$ will be almost exactly the same, no matter what.