The following propositions, I think, are generally considered as true ( though they may not all have the same level of rigor).
(1) In a set there is no order ( due to the extensionality axion) : $\{a,b\} = \{b,a\}$
(2) In an ordered pair there is an order : $(a, b)$ is not equal to $(b,a)$.
(3) An ordered pair is a set: $(a,b) = \{ \{a\} , \{a,b\} \}$.
Does the problem lie in proposition (2) : should one say, instead of " in an ordered pair there is an order" that " an ordered pair is an order"?
How to formulate rigorously these propositions in order to make them compatible?
You formally define $(a,b)$ to be equal to $\{\{a\}, \{a,b\}\}$.
That's the only formal definition.
The rest is our interpretation. We notice that using the definition above, $(a,b)$ is not equal to $(b,a)$ if $a\neq b$ (*), and therefore, in the newly introduced symbol $(.,.)$, order matters. That's why we call this newly introduced symbol an "ordered pair".
In other words, (2) is not a proposition, it is a consequence of (3).
(*) Note that the only thing you need to prove $(a,b)\neq (b,a)$ is that $a\neq b$. This is because, if $a\neq b$, then the set $\{a\}$ is an element of $(a,b)$ (because $(a,b)=\{\{a\}, \{a,b\}\}$, but it is not an element of $(b,a)$ (because $\{a\}\neq \{b\}$ and $\{a\}\neq\{b,a\}$ and there are no other elements in $(b,a)$.