My question arises from this definition: Poincare group is the group of Minkowski space-time isometries. Which means that it leaves the space-time intervals unchanged.
Now here is my understanding: Isometry is a (diffeomorphic) map (from a manifold to itself) that preserves the metric (which accepts two vectors as the input). So the phrase "space-time interval" regards the space-time (which before anything is a manifold) as a vector space. So, here, a manifold is regarded as a vector space. And I understand that $\mathbb{R}^n$ can also be regarded as a vector space.
Now my question is: what is the general setting in which a manifold can be regarded as a vector space?
$\newcommand{\Reals}{\mathbf{R}}\newcommand{\Brak}[1]{\langle #1 \rangle}$Strictly speaking, manifolds and vector spaces are "different types of fruit": Sets equipped with extra structure in such a way that neither structure naturally determines the other.
Further, the term isometry gets used without qualification in multiple settings, including (but certainly not limited to):
1. (a) A (usually real or complex) vector space $(V, \|\ \|)$ equipped with a norm or an inner product (which induces a norm). In this situation, an isometry is a linear isomorphism $i:V \to V$ that preserves the norm $\|\ \|$ in the sense that $\|i(v)\| = \|v\|$ for all $v$ in $V$.
1. (b) A (usually real or complex) affine space $(V, \|\ \|)$ equipped with a norm. In this situation, an isometry is an affine isomorphism $T:V \to V$ that preserves distances in the sense that $\|T(v) - T(w)\| = \|v - w\|$ for all $v$, $w$ in $V$.
1. (c) A (usually real or complex) vector space $(V, \Brak{\ ,\ })$ equipped with a non-degenerate bilinear pairing. In this situation, an isometry is an linear isomorphism $T:V \to V$ that preserves the pairing in the sense that $\Brak{T(v), T(w)} = \Brak{v, w}$ for all $v$, $w$ in $V$.
2. A (pseudo-)Riemannian manifold $(M, g)$. In this setting, an isometry is a diffeomorphism $\phi:M \to M$ satisfying $\phi^{*}g = g$.
In your setting these frameworks overlap: Your manifold is the real affine space $V = \Reals^{4}$ equipped with the quadratic form $$ Q = \pm(-dt^{2} + dx^{2} + dy^{2} + dz^{2}). $$ This affine space admits the structure of a smooth $4$-manifold equipped with a pseudo-Riemannian metric $g$ whose quadratic form at each point is $Q$. The Poincaré group is the group of affine isometries, which coincides with the group of isometries of the pseudo-Riemannian structure.
It's arguably more accurate to say, "here, a (real, finite-dimensional) vector space (equipped with a non-degenerate quadratic form) is regarded as a (pseudo-Riemannian) manifold" than to think of a manifold being regarded as a vector space. This interpretation is general in the sense that every finite-dimensional real vector space equipped with a non-degenerate quadratic form admits a unique smooth manifold structure for which the vector space operations are smooth, and the non-degenerate pairing defines a pseudo-Riemannian structure. (In the other direction you have to be careful. For example, there exist smooth manifolds homeomorphic to $\Reals^{4}$ but not diffeomorphic to the vector space $\Reals^{4}$.)