We cannot include $\infty$ into $\mathbb R$ if we define $0\infty =1$. However, we don't have to do so. We just define another element $1/\infty\neq 0$ to be the reciprocal of $\infty$. In that way, can we get a consistent system of numbers?
Please read the following carefully, because this is NOT the usual way we treat $\infty$. I know perfectly why we cannot define $0\infty =1$ with ordinary rules for addition and multiplication. I am NOT doing that. Please do not downvote without reading this carefully.
To be more specific, we can do arithmetic operations with this extended set of real numbers and $\infty$ and $1/\infty$. For example, $(3\times \infty)^{-1} =\frac13 1/\infty$
Is there a branch of math about this?
Sure you can get a consistent number system out of this. You have might even have used one without realizing.
Possibly the most common example is the field of rational functions (fractions where the numerators and denominators are polynomials) with real coefficients. We have an ordering on this as well: $f>g$ is defined to mean $\lim_{x\to\infty}(f-g)>0$.
The real numbers are contained in there as the constant functions, but in addition to the real numbers you have infinite elements like $x$ ("infinite" as in larger than any element of the form $1+1+\cdots +1$), and infinitesimal elements like $\frac1x$.
And here we do indeed have $(3x)^{-1} = \frac13\cdot \frac1x$, like your example says.
There are more ways of extending the real numbers to a number system with infinities and infinitesimals too, like the hyperreal numbers, or the surreal numbers, and certainly others.
If you want it to belong to a branch of mathematics, then possibly "nonstandard analysis" is the name to look for.