Solve
$$\begin{array}{ll} \text{minimize} & x^2 + y^2\\ \text{subject to} & x \neq y\\ & x + y = 2\end{array}$$
The solution does not exist; however, in practice, we can say that the solution is something like $x=1+\epsilon$ and $y=1-\epsilon$.
Of course, this explanation is not mathematically rigorous. SO my question is, how to rigorously write out this question and answer? Are there any related books that I can read?
As you observe, the minimum does not exist, because no matter which $(x,y)$ you pick, there is always a better one (by moving closer to the line $y=x$). If you want to speak of minimum "in the limit," you can speak of the greatest lower bound of $x^2+y^2$, which mathematicians sometimes call infimum, written $\inf$ instead of $\min$.