What is the estimated value of $\theta_\lambda$?

Question

Suppose we have data $D$ for $(x,y)$ where $D =\{(x_i,y_i)\} = \{(1,1),(a,b)\}$ and we have a estimator $$y=f(x;\theta)=(x;1)\theta=\theta_1x_1+\theta_2$$ for data. I want to minimize $\theta$, so I use the least squares for this. So, I have to compute the argmin for $\theta$ $$\theta_\lambda=argmin_\theta \frac{1}{2} \sum_{i=1} ^2 (y_i-f(x_i;\theta))^2 +\frac{\lambda}{2}(\theta_1 ^2 + \theta_2 ^2) ~~(\lambda ≥ 0)$$ But how should I compute this? I'm sorry but I'm completely stuck in this. Could you help me to solve this? thank you

Answer 1

Guide:

We want to find $\theta_{\lambda}$ that minimizes

$$\frac12 \sum_{i=1}^2 (y_i - \theta_1 x_i - \theta_2)^2 + \frac{\lambda}2(\theta_1^2+\theta_2^2)$$

Differentiate with respect to $\theta_1$ and equate it to zero gives me

$$-\sum_{i=1}^2 (y_i-\theta_1x_i-\theta_2)x_i + \lambda\theta_1=0$$

Similarly for $\theta_2$.

By now, you have two linear equations in two variables. You can use elimination, substitution, or some matrix trick to solve for $\theta$.

Edit:

$$-\sum_{i=1}^2 (y_i-\theta_1x_i-\theta_2)x_i + \lambda\theta_1=0$$

$$\theta_1\left(\sum_{i=1}^2x_i^2+\lambda\right)+\theta_2\sum_{i=1}^2x_i= \sum_{i=1}^2 x_iy_i$$

Similarly, differentiating the objective function with respect to $\theta_2$, we have

$$-\sum_{i=1}^2 (y_i-\theta_1x_i-\theta_2)+ \lambda \theta_2 = 0$$

$$\theta_1 \sum_{i=1}^2 x_i + \theta_2 ( n+\lambda)=\sum_{i=1}^2y_i$$

$$\begin{bmatrix} \sum_{i=1}^2 x_i^2+\lambda& \sum_{i=1}^2 x_i\\ \sum_{i=1}^2 x_i& n + \lambda\end{bmatrix} \begin{bmatrix} \theta_1 \\ \theta_2\end{bmatrix}= \begin{bmatrix} \sum_{i=1}^2 x_iy_i \\ \sum_{i=1}^2 y_i\end{bmatrix}$$

$$\begin{bmatrix} 1+a^2+\lambda& 1+a \\ 1+a& 2 + \lambda\end{bmatrix} \begin{bmatrix} \theta_1 \\ \theta_2\end{bmatrix}= \begin{bmatrix} a+b \\ 1+b\end{bmatrix}$$

$$\begin{bmatrix} \theta_1 \\ \theta_2\end{bmatrix}= \frac1{(1+a^2+\lambda)(2+\lambda)-(1+a)^2}\begin{bmatrix} 2 + \lambda & -(1+a) \\ -(1+a)& 1+a^2+\lambda\end{bmatrix} \begin{bmatrix} a+b \\ 1+b\end{bmatrix}$$