I need to find a linear mapping from the $xy$ plane to the $x'y'$ plane such that the respective coordinates are related like this:
they remain in place if the coordinate are equal $(x',y')=(x,y)$, if $x=y$
if $x \neq y$ Both $(x,y)$ and $(y,x)$ must be mapped to $(x,y)$ with $y \geq x$ so the mapping should do this point by point, "glueing together" the pairs $(x,y)$ and $(y,x)$. $(x',y')=F(x,y)=(f1(x,y),f2(x,y))$, what could $f1$ and $f2$ be and how to figure it out? Any idea?

The map you are describing is uniquely defined, so all you need to do is write it down. There are several ways you can do this, but they will all describe the same map.
Which of the three options is the best? In my honest opinion, none of them. Each is best for a particular scenario.
The first is great if you are talking about the map $F$. With no annoying symbols to decipher, it is the description that needs the least amount of time for people to understand, and is also the only one that will not scare away a non-mathematical crowd.
The second is great because it is easy to see what is happening, to a mind accustomed to some mathematical symbols.
The third is great because it's shortest, but to me, it takes a little more time to understand what exactly the map does.
How do you get to these descriptions? Well, you got to the first description just fine. The second is just writing the same thing as the first, but in symbols.
The third, well, the map $F$ takes $(x,y)$, and flips the members of the pair if $x$ is greater than $y$. In other words, it always returns the smaller of the values $(x,y)$ in the first position, and the bigger of the two values on the second position. Well, "the smaller of the two values" is simply $\min(x,y)$, and the bigger is $\max(x,y)$...