Is it possible to define determinant as canonical projection from general linear group to its abelianization? Using determinant we can show, that abelianization of $GL(n,R)$ is isomorphic to $R^{*}$ - multiplicative group of $R$ (commutative ring). I wonder, if we can go opposite way to define determinant. So question is - how to find isomorphism $\phi:GL(n, R)/\langle t_{i,j}(x)\rangle \ \cong \ R^{*}$ (where $t_{i,j}(x)$ - are elementary transvections), without using determinant? I guess, after it wouldn't be problem to show that $\phi \circ \psi$ is multilinear and antisymmetric (with projection $\psi:GL(n, R) \rightarrow GL(n, R)/\langle t_{i,j}(x)\rangle$).
Could you recommend some books, where considered abelianization of $GL(n,R)$, and related topics?
No. The abelianization map takes values in an abstract group, namely the abelianization $GL_n(R)/[GL_n(R), GL_n(R)]$. The determinant takes values in a specific concrete group, namely $R^{\times}$, and knowing this is strictly more information than just knowing the abelianization map. E.g. if $R = \mathbb{R}$ the real-valued determinant doesn't take values in the abstract monstrosity $GL_n(\mathbb{R})/[GL_n(\mathbb{R}), GL_n(\mathbb{R})]$, it takes values in the very concrete $\mathbb{R}^{\times}$, so its output is literally a number, with digits and everything.
I also don't believe it's true that the determinant is the abelianization map for general $n$ and $R$. For $R$ a field it's not true when $n = 2, R = \mathbb{F}_2$ but I expect that for $R$ not a field there should be more counterexamples. This is not a good way to think about the determinant.