Explanation of a convex quadratic program

Question

I want to find the minimum of the following optimization problem but don't even understand the problem in the first place.

$$\min\limits_{AX=b}\frac{1}{2}X^{T}QX+C^TX+\alpha$$

where $Q, A \in M_{n \times n} (\Bbb{R})$, $X, b, C \in \Bbb{R^n}$ and $\alpha \in \Bbb{R}$. Also, matrix $Q$ is symmetric and positive definite.

This is all I have about the problem. So, can anyone please, explain the problem to me? Thanks for your time.

Answer 1

You are given two fixed $n \times n$ matrices $Q$ and $A$, two fixed n-dimensional vectors $B$ and $C$, and a fixed real number $\alpha$. You are supposed to minimize the value of the objective function $f(X)=\tfrac12 X^T Q X + B^T X + \alpha$ by varying $X$, subject to the constraint $AX=B$.

So, if we define $S = \{X \in \mathbb{R}^n : AX=B\}$, then you need to find $\bar X \in S$ Such that $f(\bar X) \le f(X)$ for all $X \in S$.

So, this is a quadratic programming problem with a set of linear equality constraints.

Answer 2

You were already provided an accepted answer, but I will still attempt to provide more information. I assume that $A \in \mathbb{R}^{m \times n}$ where $m < n$, that is, the set $S = \{ X \in \mathbb{R}^n : A X = B \}$ is infinite (otherwise, the problem is trivial). Note, that the set $S$ is affine, and therefore convex.

First, observe that since $Q$ is positive definite, the quadratic objective function is coercive, and therefore a minimizer exists. Since it is strictly convex (the Hessian is $2Q$, which is positive definite), the minimizer is unique.

Now, you have several strategies. The first one, is feeding your problem into any Quadratic Programming solver software like the first answer suggests, and it will give you the (unique) solution. The major drawback of this approach is that such solvers are not always fast. But if it is good enough for you, you do not need to do anything else. Below are two other strategies.

Via KKT conditions

The convex programming problem you seek to solve trivially satisfies Slater's condition, since there are no inequality constraints. Thus, any point is optimal if and only if it satisfies the KKT conditions.

In your case, the Lagrangian is $$L(X; \lambda) = X^T Q X + C^T X + \alpha + \lambda^T (A X - B),$$ and therefore the KKT conditions are $$ 2 Q X + C + A^T \lambda = 0, \quad A X = B, $$ or equivalently written $$ \begin{bmatrix} 2 Q & A^T \\ A & 0 \end{bmatrix} \begin{bmatrix} X \\ \lambda \end{bmatrix} = \begin{bmatrix} -C \\ B \end{bmatrix}. $$ This is a linear system of equations. Any solution of this system using your favorite linear solver gives you an optimal solution of your desired optimization problem. If you can exploit the special structure of $Q$ or $A$ to solve it quickly, you can solve the optimization problem quickly.

By constraint elimination

Recall, that any rectangular matrix $V$ with more rows than columns admists a QR decomposition into $V = Q R$, where $Q$ is orthogonal and $R$ is upper-triangular. There are also plenty of numerical libraries which can compute it. Since you already used the name $Q$, I will rename your $Q$ matrix into $P$.

By computing the decomposition of $A^T$, you have $A^T = Q R$, and the problem becomes $$ \min_X X^T P X + C^T X + \alpha \quad \text{s.t.} \quad R^T Q^T X = B $$ Substituting $Q^T X = Y$, or $X = Q Y$ (since $Q$ is orthogonal), we obtain $$ \min_Y Y^T (Q^T P Q) Y + (Q^T C) Y + \alpha \quad \text{s.t.} \quad R^T X = B $$ Finally, $R$ is upper triangular. Meaning that it has the following structure $$ R = \begin{bmatrix} \tilde{R} \\ 0 \end{bmatrix}, $$ where $\tilde{R}$ is upper triangular and invertible, which is followed by rows full of zeros. Therefore, by decomposing $Y = \begin{bmatrix}Y_1 \\ Y_2 \end{bmatrix}$, where $Y_1$ are the components which correspond to the columns of $\tilde{R}$, the constraint $R^T Y = B$ can be written as $\tilde{R}^T Y_1 = B$. You can solve this system, find the value of $Y_1$, substitute it into the objective, and obtain an unconstrained problem with $Y_1$, which can be solved any off-the-shelf least-squares solver.

This approach is useful if you need to solve many problems which various matrices $P$, but all share the same matrix $A$. Moreover, it is useful if the dimension of the remaining variable $Y_2$ is much smaller than the dimension of $X$. Constraint elimination allows you to reduce the dimension of the given problem substantially in that case.