derivative with respect to constant (Lagrange multiplier) .

Question

I cannot understand how if a Lagrange multiplier is a scalar (meaning it is a constant value) that you can take a partial derivative of a Lagrangian function with respect to a constant (the Lagrange multiplier) when you need to find the gradient of a Lagrangian function.

If the derivative of a constant is zero, then how/why is it even possible to take the derivative of something with respect to a constant/Lagrange multiplier?

Answer 1

It's common to call something a "constant" in informal descriptions of a math problem, but it's important to always keep in mind: constant with respect to what?

Let's look at a concrete example: linear-constrained least squares,

$$\min_{\mathbf{x}}\ \sum_{i=1}^{n} (x_i - y_i)^2 \quad \mathrm{s.t.} \quad \sum_{i=1}^n x_i = b.$$ Here the vector $\mathbf{x}\in \mathbb{R}^n$ is the independent variable of the optimization problem, and $y_i$ and $b$ are constants with respect to $\mathbf{x}$.

We can reformulate this problem using the method of Lagrange multipliers: $$\underset{\mathbf{x},\lambda}{\operatorname{ext}} \ \sum_{i=1}^{n} (x_i - y_i)^2 - \lambda\left(\sum_{i=1}^n x_i - b\right).$$ In this new problem:

• $\lambda$ does not depend on $i$; it's a single "global constant";
• $\lambda$ is constant with respect to $\mathbf{x}$; it's an independent variable, and not a function of $\mathbf{x}$;
• $\lambda$ is an independent variable (not a constant) of the optimization problem: you are trying to solve for $\lambda$ as part of the optimization problem, and are not given the value of $\lambda$ in advance.

Notice that there are three different notions of being "a constant" here and whether or not $\lambda$ is a constant depends on which you mean.