Problem: Given $((x_{11},...,x_{1M},y_1),...,(x_{N1},...,x_{NM},y_N))$ and the weights $w_1,...,w_N$ , where for every $n$ is true that $w_n>0$ . Find the $θ_0,θ_1,...,θ_M$ which minize the following error function:
$$J(θ_0,θ_1,...,θ_M)=\sum_{n=1}^{N}w_n\left(y_n-θ_0-\sum_{m=1}^{M}θ_mx_{nm}\right)^2$$
Let us evaluate $-\frac{1}{2}\frac{\text{d}J}{\text{d}\theta_i},~i=0,\dots,M$ to make expressions easier: \begin{align} -\frac{1}{2}\frac{\text{d}J}{\text{d}\theta_0}&=\sum_{n=1}^{N}w_n\left(y_n-θ_0-\sum_{m=1}^{M}θ_mx_{nm}\right)=0,\\ -\frac{1}{2}\frac{\text{d}J}{\text{d}\theta_i}&=\sum_{n=1}^{N}w_n\left(y_n-θ_0-\sum_{m=1}^{M}θ_mx_{nm}\right)x_{ni}=0,~i=1,\dots,M. \end{align} After rearranging the above equations we get the linear system \begin{align} \sum_{n=1}^{N}w_n\theta_0+\sum_{j=1}^M\left[\sum_{n=1}^{N}w_nx_{nj}\right]\theta_j&=\sum_{n=1}^{N}w_n y_n \\ \sum_{n=1}^{N}w_n\theta_0+\sum_{j=1}^M\left[\sum_{n=1}^{N}w_nx_{nj}x_{ni}\right]\theta_j&=\sum_{n=1}^{N}w_n y_n x_{ni},~i=1,\dots,M. \\ \end{align} The system can be written as follows: \begin{align} \left[\begin{array}{cccc} \sum_{n=1}^{N}w_n & \sum_{n=1}^{N}w_n x_{n1} & \dots & \sum_{n=1}^{N}w_n x_{nM}\\ \sum_{n=1}^{N}w_n & \sum_{n=1}^{N}w_n x_{n1}x_{n1} & \dots & \sum_{n=1}^{N}w_n x_{n1}x_{nM} \\ \vdots & \vdots & & \vdots \\ \sum_{n=1}^{N}w_n & \sum_{n=1}^{N}w_n x_{nM}x_{n1} & \dots & \sum_{n=M}^{N}w_n x_{n1}x_{nM} \\ \end{array}\right] \left[\begin{array}{c} \theta_0 \\ \theta_1 \\ \vdots \\ \theta_M \end{array}\right] = \left[\begin{array}{c} \sum_{n=1}^{N}w_n y_n \\ \sum_{n=1}^{N}w_n y_nx_{n1} \\ \vdots \\ \sum_{n=1}^{N}w_n y_nx_{nM} \end{array}\right] \end{align} which is a linear system of equations of the form $A\mathbf{\theta}=b$.
Let us further show that the least square functionals are convex. In general the functional can be written in the form $J(x)=\sum_i (A_ix-b_i)^2$. Let us show that $J(x)$ is convex. Let $x\neq y$. We show that for any convex combination $J\left(\theta x+(1-\theta)y\right)\le\theta J(x)+(1-\theta)J(y)$. And really: \begin{align} &J\left(\theta x+(1-\theta)y\right)=\sum_i \left[\theta(A_ix-b_i)+(1-\theta)(A_iy-b_i)\right]^2=\nonumber\\ &\sum_i\left[\theta^2(A_ix-b_i)^2+2\theta(1-\theta)(A_ix-b_i)(A_iy-b_i)+(1-\theta)^2(A_iy-b_i)^2\right]\equiv\nonumber\\ &\sum_i\left[\theta(A_ix-b_i)^2+(1-\theta)(A_iy-b_i)^2\underbrace{-\theta(1-\theta)\left[A_i(x-y)\right]^2}_{\le0}\right]\le\nonumber\\ &\sum_i\left[\theta(A_ix-b_i)^2+(1-\theta)(A_iy-b_i)^2\right]=\theta J(x)+(1-\theta)J(y).~\spadesuit \nonumber \end{align}