So, when we solve the optimization problem using Lagrange Multiplier method, I know that lambda can be positive or negative. Lambda is simply the rate of change in the optimal value when the constraint changes. So, I understand that lambda can be positive or negative.
Now, my question is when we have inequality constraints. In that case, why do we have the requirement that lambda should be non-negative? I could understand the math behind it. But I am not able to follow up the intuition. Simply put, if Lambda is still the rate of change with respect to the change in the constraint, then why THEORETICALLY SPEAKING can't it be negative?
It is only under certain circumstances that the Lagrange multipliers represent the sensitivity of the optimal cost to a change in a constraint bound.
Different conventions used for the multipliers. In the context of a problem $\min \{ f(x) \mid g_k(x) \le b_k \}$, the convention I prefer is that at a local $\min$ (assuming a suitable constraint qualification is satisfied) there are non negative multipliers $\lambda_k$ such that $\nabla f(x) + \sum_k \lambda_k \nabla g_k(x) = 0$ (plus complimentary slackness, etc.).
To simplify life, assume there is just one constraint $g(x) \le b$. Under lots of extra conditions, we have that the sensitivity of the optimal cost to $b$ is given by $- \lambda$. That is, the sensitivity is non positive.
To see why this makes sense, if you increase $b$, the feasible set gets larger and so the resulting $\min$ will be lower (or at least, not larger), hence the sensitivity is negative (well, non positive).
In the context of an equality constraint $h(x) = b$, note that this is exactly the same as the constraint $-h(x) = -b$, so there is no 'preferred' sign as is the case with inequality constraints. Also, note that the feasible set does not have the same containment (as in $\subset$) relationship as with inequality constraints, so one cannot expect a particular sign.