There are two agents $i \in \{1,2\}$ who share a dollar. The amount agent $i$ receives is denoted with $x_i \in [0,1]$. The set of feasible allocations is then given by by $X = \{(x_1,x_2) \mid x_1 + x_2 \leq 1\}$. An agent receives utility according to the function $u_i(x_i) = \sqrt{x_i}$. Note that $u_i(x_i) \in [0,1]$ for all $x_i \in [0,1]$. Define the set of feasible outcomes by $S = \{(u_1(x_1),u_2(x_2)) \mid (x_1,x_2) \in X\}$. For $S$ being a well-defined bargaining problem we need to show that $S$ is convex and compact.
Convexity We show that $S$ is compact, by showing that the Pareto frontier is concave. The Pareto frontier is defined by efficient (no waste) allocations $P = \{(u_1(x_1),u_2(x_2)) \mid x_1 + x_2 = 1\}$. The Pareto frontier is thus the boundary of $S$, i.e., $P \subset \partial S$. The inverse of $u_1$ is given by $x_1(u_1) = u_1^2$. With $x_1 + x_2 = 1$ it is easy to see that the Pareto frontier is given by the following funtion $u_2(u_1) = \sqrt{1-u_1^2}$. Since $u''_2(u_1) < 0$ for all $u_1 \in [0,1]$, $S$ is convex.
I was wondering how to show convexity from the primitive definition $\lambda (u_1(y_1),u_2(y_2)) + (1-\lambda)(u_1(z_1),u_2(z_2)) \in S$ for all $y,z \in X$ with $\lambda \in [0,1]$.
I also do not know how one shows compactness.
Convexity: Take arbitrary $y, z \in X$ and $\lambda \in [0, 1]$. We need to find a point $x \in X$ such that $$ \begin{pmatrix} \lambda u_{1}(y_{1}) + (1 - \lambda) u_{1}(z_{1}) \\ \lambda u_{2}(y_{2}) + (1 - \lambda) u_{2}(z_{2}) \end{pmatrix} = \begin{pmatrix} u_{1}(x_{1}) \\ u_{2}(x_{2}) \end{pmatrix} $$ Our candidate point is $(x_{1}, x_{2})$ with $x_{1} = u_{1}^{-1}\left(\lambda u_{1}(y_{1}) + (1 - \lambda) u_{1}(z_{1})\right)$ and $x_{2} = u_{2}^{-1}\left(\lambda u_{2}(y_{2}) + (1 - \lambda) u_{2}(z_{2})\right)$. We only have to show that $x_{1} + x_{2} \leq 1$, as otherwise $x$ does not belong to $X$. This is where concavity of $u_{1}$ and $u_{2}$ comes in: By Jensen's inequality and the fact that $u_{1}$ is increasing, $$ x_{1} = u_{1}^{-1}\left(\lambda u_{1}(y_{1}) + (1 - \lambda) u_{1}(z_{1})\right) \leq u_{1}^{-1}\left(u_{1}\left(\lambda y_{1} + (1 - \lambda) z_{1} \right)\right) = \lambda y_{1} + (1 - \lambda) z_{1}. $$ A similar argument shows $$ x_{2} \leq \lambda y_{2} + (1 - \lambda) z_{2}. $$ Therefore, $x_{1} + x_{2} \leq \lambda (y_{1} + y_{2}) + (1 - \lambda) (z_{1} + z_{2}) \leq \lambda + (1 - \lambda) = 1$, where the inequality uses that $y$ and $z$ belong to $X$ by assumption.
Compactness: Here, use the fact that continuous functions map compact sets to compact sets. The set $S$ is precisely the image of $X$ under a continuous function.