I was reading the book The ABC of Relativity from Betrand Russell, and at some point, the author mentions a method for measuring the distance between 2 points on any coordinate system. He says that the formula was discovered by Gauss and is a generalization of the pythagorean theorem. Although, it doesn't writes the explicit formula in mathematics. Since I have no more information, couldn't find anything on it. If I understand the formula correctly, it should be something like this when the coordinates can be expressed with 2 components $(x;y)$:
$$d(A,B)^2= (x_A-x_B)^2+(y_A-y_B)^2+(x_A-x_B)(y_A-y_B)$$
Is this formula correct and where it comes from?
I'm unsure if my answer will cover what you want, but since you've found this statement in a book of Relativity this probably is in the context of differential geometry. Well, the idea is that you can generalize the notion of distance to any manifold you like by introducing the notion of a Riemannian Metric and the notion of a Geodesic. When I first learned General Relativity the books were a little confusing about that so I understand your worry.
Well, what are all of these and what the hell they have to do with Gauss? Well, first, manifold is a generaliztion of surface. Gauss studied surfaces in $\mathbb{R}^3$, however, mathematicians saw that the results obtained didn't depend on the structure of the background euclidean space, this leads to a generalization to a much more general object called manifold, which in simplest terms is one space on which we can do geometry as we do on surfaces. A manifold is characterized by the nice property that each of it's points has a neighbourhood equivalent in some sense to the euclidean space.
The classical way to tackle the problem of distances in surfaces is to introduce the object called the first fundamental form. The idea is that at each point $p$ of the surface $S$ we plug the tangent plane $T_pS$ and the elements of the tangent plane are the tangent vectors to $S$ at the point $p$ (imagine this, is not hard). We will be able to measure distances if we introduce on each of the tangent planes one inner product. Since the surface is a subset of $\mathbb{R}^3$ we already have one candidate to the inner product: we just restrict that of $\mathbb{R}^3$. Then if $v,w\in T_pS$ let's denote $\left\langle v, w\right\rangle_p$ the inner product of $\mathbb{R}^3$ restricted to $T_pS$. Now we introduce the first fundamental form as $I_p : T_pS \to \mathbb{R}$ by requiring that:
$$I_p(w)=\left\langle w, w\right\rangle_p$$
Now look at what we did: in $\mathbb{R}^3$ alone the inner product allows us to compute the length of vectors by taking the square root of the inner product of the vector and itself. Now the tangent plane is a subset of $\mathbb{R}^3$ so we can restric the inner product to the tangent plane and use it's inner product to define that quadratic form. That quadratic form gives us the notion of length of vectors in the tangent plane. Now if you recall what is the length of a curve and how do we compute it you'll see that introducing this quadratic form we are allowed to compute the length of curves over surfaces.
But think for a while, in three-space from all possible curves joining two points $A$ and $B$, which of them we select the length to call distance from $A$ to $B$? Well, we select that that gives us the minimum length which is the line segmenet joing the points. So in a surface something similar happens, we define a curve in the surface that is the analog of a line in a surface, this curve we call a geodesic, and when we pick a geodesic between points we call the length of the geodesic the distance between the points.
You might say: that's all cool, but I want General Relativity! It's a four dimensional space time not embedded into any other higher dimensional space! And what I tell you is: calm down, it is all related. The problem is that when you remove the background space you have to be more careful, you'll need to introduce the notion of a manifold, and since we don't have the background space we'll have to refine the notion of a tangent space and of the inner product. In practice, the inner product association to each point will become a tensor field called the metric tensor (which is the term $g_{ab}$ which appears in Einstein's Equation).
And finally how do you come to learn all of this? Well, my recommendation is: to get started study Do Carmo's Differential Geometry of Curves and Surfaces - it's a book about the differential geometry of surfaces in $\mathbb{R}^3$, good to get started. Then you'll need to move to the differential geometry of manifolds. My recommendation is that you pick Spivak's A Comprehensive Introduction to Differential Geometry. Spivak however uses the notion of metric spaces, if you don't know anything about them you can go after a book about metric spaces, and to study this my recommendation is to get Rudin's Principles of Mathematical Analysis and read the chapter of metric spaces and after it get some topology book and read the chapter of metric spaces.
I hope this helps you somehow. Good luck!