According to Nitsure's Construction of Hilbert and Quot Schemes (p. 16), if $S$ is Noetherian, a \textit{linear scheme over $S$} is a scheme of the form $V=\operatorname{Spec}\operatorname{Sym}_{\mathsf{O}_{S}}\mathsf{Q}$ for some coherent sheaf $\mathsf{Q}$ over $S$.
The author makes the claim that this is naturally a group scheme. Why?
Also, how does this generalize vector bundles?
On
The answer to your first question is because $V$ represents the group-valued functor taking an $S$-scheme $f: T \to S$ to $(f^* Q^\vee)(T)$, where $Q^\vee$ is the dual of $Q$.
If $Q$ is a locally free $\mathcal{O}_S$-module of rank $n$, then $V$ is locally (over an open cover $U_i$ of $S$) of the form $U_i \times \mathbb{A}^n$, hence a vector bundle.
For reference see EGA II sections 1.3 and 1.7, Stacks, or section 17.1 of Vakil. There's also an exercise about this in Hartshorne.
On
Question: "The author makes the claim that this is naturally a group scheme. Why? Also, how does this generalize vector bundles?"
Answer: If $V:=k\{e\}$ is a rank one free module over a commutative ring $k$ and $S:=Sym_k^*(V^*)\cong k[x]$ is the polynomial ring on $x$ over $k$, there is a canonical map
$$m:k[x]\rightarrow k[x]\otimes_k k[x]\cong k[x,y]$$
defined by $m(x):=x+y$ and $m(f(x)):=f(x+y)$. This defines a $k$-group scheme structure on $\mathbb{A}^1_k:=Spec(S)$. If $k$ is a field, this group scheme structure corresponds to the fact that the $k$-vector space $V$ has an underlying additive abelian group: There is an addition operation
$$+: V \times V \rightarrow V$$
defined by $+(u,v):=u+v$ making $(V,+)$ an abelian group. The $k$-group scheme $\mathbb{A}^1_k$ is a "scheme version" of this structure. This generalize to any scheme $X$ and any algebraic vector bundle $E$ on $X$: We define $\mathbb{V}(E^*):=Spec(T)$ where $T:=Sym_{\mathcal{O}_X}^*(E^*)$ is the "sheaf symmetric algebra" of $E^*$ and take relative Spec of this. The sheaf $T$ is a sheaf of commutative unital $\mathcal{O}_X$-algebras and you may take the "relative Spec" of this sheaf to get a scheme
$$\pi: \mathbb{V}(E^*)\rightarrow X.$$
The group scheme $\mathbb{V}(E^*)$ is the geometric vector bundle of the finite rank locally trivial sheaf $E$. The map $\pi$ has the property that for any point (closed or not) $x\in X$ there is a canonical isomorphism
$$\pi^{-1}(x) \cong \mathbb{A}^e_{\kappa(x)}$$
where $e=rk(E)$ and $\kappa(x)$ is the residue field of $x$.
This is a consequence of the fact that the symmetric algebra of $\mathcal{Q}$ naturally forms a cogroup object in the category of $\mathcal{O}_S$-algebras. Namely, this cogroup object has comultiplication map $S^\cdot \mathcal{Q} \to S^\cdot \mathcal{Q} \otimes_{\mathcal{O}_S} S^\cdot \mathcal{Q}$ which is determined by the requirement that for sections $x \in \mathcal{Q}(U)$, $x \mapsto 1 \otimes x + x \otimes 1$. Similarly, the coinverse map $S^\cdot \mathcal{Q} \to S^\cdot \mathcal{Q}$ is the map such that for $x \in \mathcal{Q}(U)$, $x \mapsto -x$; and the coidentity map $S^\cdot \mathcal{Q} \to \mathcal{O}_S$ is the map such that for $x \in \mathcal{Q}(U)$, $x \mapsto 0$. It it straightforward to show that these definitions satisfy the required identities for a cogroup object. And then, taking $\mathbf{Spec}$ of a cogroup object will result in a group object in the category of $S$-schemes since $\mathbf{Spec}$ is contravariant and takes coproducts to products.
As for how to see this as a generalization of vector bundles, the answer by ggg already gives one possibility: if $\mathcal{Q}$ is locally free then $\mathop{\mathbf{Spec}} S^\cdot \mathcal{Q}$ is in fact a vector bundle. Another possibility: you can show that in fact, $\mathop{\mathbf{Spec}} S^\cdot \mathcal{Q}$ is a module over the ring object $\mathbb{A}^1_S$ using a similar construction of $S^\cdot \mathcal{Q}$ as a comodule over the coring object $\mathcal{O}_S[t]$. Thus, for example if $S$ is a $k$-scheme where $k$ is a field, then every element of $k$ induces a section $S \to \mathbb{A}^1_S$, and using the module structure this induces an endomorphism of $\mathop{\mathbf{Spec}} S^\cdot \mathcal{Q}$; it is then not hard to see that this gives every fiber of $\mathop{\mathbf{Spec}} S^\cdot \mathcal{Q}$ a structure of $k$-module object. So, for example, the closed points of each fiber form a $k$-vector space.