If I define a binary relation $r$ simply like a set of pairs, in formal way:
$\forall u ( u\in r \rightarrow \exists x\exists y\ u=(x,y))$
Is it possible to prove that the class $dom(r)=\{x|\exists y\ (x,y)\in r\}$ is a set?
I think that is a wrong definition and the right formal definition is:
$\exists a \exists b \forall u ( u\in r \rightarrow \exists x \in a \exists y \in b \land \ u=(x,y))$
You ask (paraphrased):
The answer is: yes, if by "binary relation" we mean "set binary relation." There are two kinds of relations: set relations, which are sets of ordered pairs, and class relations, which are classes of ordered pairs. A class relation need not have a set as its domain: e.g. the relation "$x=x$" is a class relation with domain all of $V$.
On the other hand, suppose $r$ is a set of ordered pairs. Then we can indeed prove that its domain is a set. One way to do this is the following:
Let $\varphi(x, y)$ be the formula "$y$ is the left coordinate of $x$" - that is, "$\exists z(x=(y, z))$."
For each $a\in r$ there is exactly one $b$ satisfying $\varphi(a, b)$; so we may apply Replacement to get $\{b: \exists a\in r(\varphi(a, b))\}$.
But this set is exactly $dom(r)$!