I would like to know if is there any theory about optimization when the objective function is not unique. For example, I would like to maximize the sum of utilities of some agents but also I would like to minimize the sum of the prices that the agents pay.
It is like find a trade-off between two or more things. Is there any optimization theory like this? Do you know any good paper that deals with such trade-off of maximizing something and minimizing another thing?
In an optimization problem you need one objective function. For example, in the usual profit maximization problem
$$\max_{x\geq 0}\pi(x)$$
where $\pi$ is our profit/objective function. Letting $R$ be the revenue function and $C$
$$\pi(x)=R(x)-C(x)$$
so maximizing profit is maximizing the sum of revenue and negative cost. More generally, we could incorporate the tradeoff between two objective using a weighted sum of the two objective functions $g$ and $h$:
$$f(x)=\lambda g(x)+(1-\lambda)h(x)$$
where $\lambda\in[0,1]$. In the profit maximization example, $\lambda=1/2$, $g(x)=2R(x)$, and $h(x)=-2C(x)$.
For the example mentioned in the OP, $g$ could be the sum of utilities, and $h$ the negative of the sum of prices paid as a function of the choice variable $x$.
But you may want to think about whether what you want to do makes sense in terms of the theory.