I was learning about covariant and contravariant vectors due to special relativity, and it occured to me that we don't live in $\mathbb{R}^4$. I'll explain myself better. Consider the space of polynomials of degree $\leq n$. This is a vector space, and choosing ${1,x,x^2,...,x^n}$ as a basis makes perfect sense: those objects exist regardless of their coordinates. However, when we set up a coordinate system in classical mechanics (for example) we have a preesisting space (the "physical world"), in which we choose a point (which is going to be the $0$ of our identification with $\mathbb{R}^3$) and then a basis with which we are able to give coordinates to that point. So far so good, except that I have no idea what the "physical world" is without resorting to an $\mathbb{R}^3$ description in the first place.
Now, I thought of thinking about the "physical world" in a manifold sense: you have a topological space $(M, \tau)$ and you construct an atlas with one chart. In that sense, we can talk about coordinates and when we choose a reference point we are equipping the topological space of a manifold structure to talk unambiguously of points.
My question is: what's that topological space? What is the space I am equipping with a manifold structure?
So, I answer the question since I've found the answer I was looking for and it might be useful to somebody else sooner or later. I'll treat the case of special relativity. Axiomatically, we assume that $M^4$ is an affine space, that is a triad $(A^n, V, \vec \cdot)$ where $A^n$ is a set whose elements are called points, $V$ is an $n$-dimensional vector space and $\vec \cdot$ is an operation with the following properties:
Now, once the origin $O$ is fixed there clearly is a bijection $f:V \rightarrow \mathbb{R}^n$. The euclidean topology naturally induces a topology on $A^n$, namely the open sets are the counterimages of the open sets of $\mathbb{R}^n$. We'll call the function $f$ "system of cartesian coordinates" of origin $O$ and axes $e_1,...,e_n$. Moreover we can equip, using this function, $A^n$ of a structure of topological and differentiable $C^\infty$ manifold. We want more structure though, and since we have a simmetrical bilinear form on $V$ ($\operatorname{diag}(1,-1,-1,-1)$) we'd like to transport it to $A^n$. Since it is a differentiable manifold we have no problem in taking the tangent spaces $T_pA^n$(which are essentially $\mathbb{R}^n$) and constructing an isomorphism $L_p: T_pA^n \rightarrow \mathbb{R}^n$ which sends the canonical basis of the tangent space in the canonical basis of $\mathbb{R}^n$. Then we define, $\forall u, v \in T_pA^n$:
$$g_p(u,v)=g(L_pu,L_pv)$$
And that's as far as I got. Please correct me if something's off.