This paper says that given a finitely generated group $\Gamma$, let $d_N(\Gamma)$ be the dimension of the space of irreducible rank $N$ representations. The number $d_1(\Gamma)$ coincides with the first Betti number, so one may think of $d_N(\Gamma)$ as a nonabelian generalisation.
Now given a compact manifold $X$, take $\Gamma$ to be $\pi_1(X)$, then the the space of irreducible rank $1$ representations is just the $Hom(\pi_1(X),\mathbb{C}^*)\cong Hom(H_1(X),\mathbb{C}^*)\cong H^1(X,\mathbb{C}^*)$. Now I wonder why $H^1(X,\mathbb{C}^*)$ is a vector space?
And $\mathbb{C}^*$ is not a ring, I wonder how to get the first Betti number?
Also what does it mean by "so one may think of $d_N(\Gamma)$ as a nonabelian generalisation"?
Why should this be a vector space? It is not. As for the statement in the paper: It is correct. The space $Hom_{irr}(\pi_1, GL(n, {\mathbb C}))$ (or a suitable quotient of this space by the action of $GL(n, {\mathbb C})$) is a complex-algebraic variety, so it has dimension. This dimension can be regarded as a generalization of the dimension of a vector space, just as the dimension of a general topological space is a far-reaching generalization of the notion of dimension from linear algebra.
Edit 1. In order to see that $Hom(\pi_1, GL(n, {\mathbb C}))$ is an algebraic variety, consider a presentation of $\pi_1$: $$ <s_1,...,s_k| R_1(s_1,..,s_k), R_2(s_1,...,s_k),....>. $$ Then $Hom(\pi_1, GL(n, {\mathbb C}))$ is identified with the subset of $GL(n, {\mathbb C})^k$ given by the equations $$ R_i(x_1,...,x_k)=1, i=1,2,... $$ and inequalities $det(x_1)\ne 0,..., det(x_k)\ne 0$, where $(x_1,...,x_k)\in Mat(n\times n, {\mathbb C})^k$.
The latter is a complex-algebraic variety (not necessarily irreducible). One can verify that the subset of irreducible representations is Zariski-open, making $Hom_{irr}(\pi_1, GL(n, {\mathbb C}))$ an algebraic variety as well. Regarding quotients by conjugation, see the reference I gave here.
Edit 2. The subset of irreducible representations is Zariski-open is proven for instance in Proposition 1.10 (page 11) in the book
Alexander Lubotzky, Andy R. Magid, Varieties of Representations of Finitely Generated Groups, Memoirs of AMS, vol. 336, 1985.
Make sure you read the discussion on the preceding page 10 as well.
Edit 3. In fact, $Hom(\pi, GL(n))$ is not just a variety but a scheme, this is again explained nicely in the reference above. The associated functor of points is $$ A\mapsto Hom(\pi, GL(n))[A]= Hom(\pi, GL(n,A)) $$ for commutative rings $A$. This scheme has a quotient, traditionally called the character scheme (topologists tend to call the quotient the character variety), which can be understood either as the GIT/Mumford quotient $$ X=X(\pi, n)=Hom(\pi, GL(n))//GL(n) $$ where $GL(n)$ acting on representations via conjugation (see the same reference) or, better as a stack. For the latter interpretation I do not have a good reference. This distinction becomes important for instance when you consider the Zariski tangent space of the character scheme at $[\rho]\in X=X(\pi,n)$: In general it is not isomorphic to $H^1(\pi, Ad\circ \rho)$. However, if one interprets $X$ as a stack, then $$ T_{[\rho]} X\cong H^1(\pi, Ad\circ \rho). $$