Linear regression with two independent variables

92 Views Asked by At

I want to consider purely theoretical situation that we have two random $\textbf{independent}$ variables $Y$ and $X$ which has $EY = EX = 0$. Let's also consider regression $Y \sim X$. I want to say what will be the coefficient $\beta$ in this regression associated with $X$ when we consider model with and without intercept:

Model with intercept

Since model has intercept we can write $\beta$ as:

$$\beta = \frac{\textrm{Cov}(X, Y)}{\textrm{Var}(X)} = 0$$

So the coefficient will be $0$, because $X, Y$ are independent (I didn't use information that $EY = EX = 0$)

Model without intercept

When we consider model without intercept, then $\beta$ is given as:

$$\beta = \frac{\sum_{i = 1}^n x_iy_i}{\sum_{i = 1}^n x_i^2}$$

Which can be rewriten as:

$$\beta = \frac{\sum_{i = 1}^n \frac{x_iy_i}{n}}{\sum_{i = 1}^n \frac{x_i^2}{n}} \rightarrow \frac{E[XY]}{E[X^2]} = \frac{EXEY}{E[X^2]} = 0$$

In second case, our $\beta$ will converge to $0$, when $n$ goes to infinity.

Is my justification correct?

1

There are 1 best solutions below

0
On

I'm not sure why you have population moments in one calculation and data in the other calculation.

In any case, for a single regressor with intercept, ordinary least squares solves

$$\min _{\alpha,\beta}\sum_{i=1}^n (Y_i-\alpha-X_i\beta)^2. $$

With intercept, the above gives $$\hat \beta_{\text{int}}=\frac{\frac{1}{n}\sum_i X_iY_i-\left(\frac{1}{n}\sum_i X_i\right)\left(\frac{1}{n}\sum_i Y_i\right)}{\frac{1}{n}\sum_i X_i^2-\left(\frac{1}{n}\sum_i X_i\right)^2}.$$

Without intercept ($\alpha=0$), the above gives

$$\hat \beta_\text{no int}=\frac{\frac{1}{n}\sum_i X_iY_i}{\frac{1}{n}\sum_i X_i^2}.$$

Assuming $(X_i,Y_i)$ are iid and that $X_i,Y_i$ have finite second moments and that $X_i$ has nonzero variance, the weak law of large numbers and continuous mapping theorem then imply

$$\hat \beta_{\text{int}}\overset{p}{\rightarrow }\frac{\text{Cov}(X_i,Y_i)}{\text{Var}(X_i)},\\ \hat \beta_\text{no int}\overset{p}{\rightarrow }\frac{E[X_iY_i]}{E[X_i^2]}.$$

Of course, these probability limits are both identically zero when $Y_i,X_i$ have mean zero and are uncorrelated, as in your setup.