Linear Least Squares

Question

I again found myself in trouble understanding the following problem (1b)

I understand linear least squares as that I have a data points, and I am trying to find a line between them which approaches given data the best and thus I want to minimaze the distance between the model and the real system. I know the formula, I know a way but I am quite clueless how to apply that.

My idea is $e[k]=y-\hat{y}$ and thus substituting that to the formula I listed above, and then using the system formula rewrite e as a function of theta and then do the partial derivation with respect to theta and so obtain a minimum.

Can anyone please give me a hint if I am thinking correctly and if so how to get rid of that norm ? Thanks a lot

Answer 1

Note that $$\hat{y}[k] = \theta u[k].$$ Consequently, $$\|y-\hat{y}\|^2 = \sum_{k=0}^{N-1}(y[k]-\theta u[k])^2.$$ In order to compute $\theta$, we take the derivative of the above expression w.r.t. $\theta$ and equate to $0$. That is, $$\frac{\partial }{\partial \theta}\sum_{k=0}^{N-1}(y[k]-\theta u[k])^2 = -2\sum_{k=0}^{N-1}(y[k]-\theta u[k])u[k] = 0 \implies \theta \sum_{k=0}^{N-1}(u[k])^2=\sum_{k=0}^{N-1}y[k]u[k].$$ Therefore, $$\hat{\theta} = \frac{\sum_{k=0}^{N-1}y[k]u[k]}{\sum_{k=0}^{N-1}(u[k])^2}.$$

Obviously, $\theta$ cannot be estimated when $u[k]=0$.

Lastly, use the fact that $E[y[k]]=E[\theta u[k] + e[k]] = \theta u[k] + 1$ to calculate the bias of $\hat{\theta}$.

Answer 2

Suppose $u[k] = 0$ for all $k$. By assumption, this means $$ y[k] = e[k] $$ Indeed, this means $$ \big[ \|\hat y - y\|_2^2\big] = \big[ \|\hat y - e\|_2^2 \big] $$ and in particular, if were minimizing in expectation, it would follow that $$ \mathbb{E} \big[ \|\hat y - y\|_2^2\big] = \mathbb{E} \big[ \|\hat y - e\|_2^2 \big] $$ If $e[k]$ were known to be iid according to $\mathcal{N}(1, \sigma^2)$, then the above expected squared error would be smallest for $\hat y = \mathbb{E}[ e ] = 1$. The calculation $\hat y = \mathbb{E} [ e]$ can be done using elementary calculus techniques and knowledge that $e[k] \sim \mathcal{N}(1, \sigma^2)$.