Linear Least Squares with Linear Equality Constraints

Question

I want to solve the least squares problem $(Ax-b)^2$ with no intercept term for linear regression with the constraint that the sum of the weights is equal to 1.

I am trying to get the closed form solution of it but couldn't. Any suggestion?

Answer 1

The problem is given by:

$$ \begin{alignat*}{3} \arg \min_{x} & \quad & \frac{1}{2} \left\| A x - b \right\|_{2}^{2} \\ \text{subject to} & \quad & \boldsymbol{1}^{T} x = 1 \end{alignat*} $$

The Lagrangian is given by:

$$ L \left( x, \nu \right) = \frac{1}{2} \left\| A x - b \right\|_{2}^{2} + \nu \left( \boldsymbol{1}^{T} x - 1 \right) $$

From KKT Conditions the optimal values of $ \hat{x}, \hat{\nu} $ obeys:

$$ \begin{bmatrix} {A}^{T} A & \boldsymbol{1} \\ \boldsymbol{1}^{T} & 0 \end{bmatrix} \begin{bmatrix} \hat{x} \\ \hat{\nu} \end{bmatrix} = \begin{bmatrix} {A}^{T} b \\ 1 \end{bmatrix} $$

Now all needed is to solve the above with any Linear System Solver.