Is there some term in math for something like reverse of relation?
So we have relation in matrix form for example:
A B C
A 0 1 1
B 1 0 0
C 1 0 0
And I want to mathematically express exactly opposite :
A B C
A 1 0 0
B 0 1 1
C 0 1 1
Is there some notation for this?
Yes. If you think of a relation as a subset $R$ of some $A\times A$, then the complementary relation is just $(A\times A)\setminus R$. If you use relation-oriented notation, then you would just write it with a slash through the relation, like $S\not\subset T$.