I call this hypotetic category BigSet. The definition of BigSet is:
- Objects are sets
- Morphisms are binary relations (including functions)
- Composition is the composition of relations
- Identities are the identity functions
Functions are special binary relations, so this category differs from Set only that in BigSet there are more morphisms than in Set. Am I right that this is a category? Is it used in mathematics? I yes, where, if not, why?
Yes, these have been considered. There is a whole research monograph around this theme, Categories and Allegories by Freyd and Scedrov.
The nLab has an entry on the category you define, which it calls Rel. (The page is written with one eye towards n-category theory, so it's a bit heavy on the terminology.)
Categories like Rel have extra properties that can be exploited. Categories with these extra properties are called allegories.