When considering the surface $S: x^2+y^2+z^2 = R^2$ we know that the surface integral
$$ \iint_S dS = 4\pi R^2$$ Since this is the area of a sphere, but while using surface integral I know that the $dS$ is the norm of the normal vector of the surface $n = (2x,2y,2z)$, $||n|| = \sqrt{4(x^2+y^2+z^2)} = 2R$, over the area of the circle it is on top of, on which it follows
$$\iint_SdS = \iint\limits_{x^2+y^2\leq R} 2RdA = 2R\cdot(\pi R^2) = 2\pi R^3$$
And I dont know why the result is diferent on an $R$ factor. The $\times 2$ might be cause the sphere is not a function and has to be split on 2 equal parts, but I dont know how I ended up with an extra $R$ term.
The formula you are referring to, where you integrate the magnitude of the normal vector, is dependent on the parameterization of the surface that you choose. You didn't specify a parameterization, which is maybe the source of your confusion.
In general, if a surface is given by $\vec{r}(u,v) = (x(u,v),y(u,v),z(u,v))$, with partial derivative vectors $\vec{r}_u = (x_u,y_u,z_u)$ and $\vec{r}_v=(x_v,y_v,z_v)$, then the normal vector you want to use is the cross product $\vec{n} = \vec{r}_u \times \vec{r}_v$. Notice that any re-scaling of a normal vector is also a normal vector, so it matters which particular normal vector you use.
There are many ways of parameterizing the sphere. One way is using "graph coordinates", where the parameterization is $\vec{r}(x,y) = (x,y,\sqrt{R^2-x^2-y^2})$. In this case, the normal vector from the formula above has length $$ \| \vec{n} \| = \frac{R}{\sqrt{R^2-x^2-y^2}} $$ Integrate this over the disc of radius $R$ (use polar coordinates!) to get $2\pi R^2$. This only gives half the sphere, so double it to get the correct answer.
Another convenient way is to parameterize using spherical coordinates. The parameterization would be $\vec{r}(\theta,\phi) = (R\sin(\phi)\cos(\theta), \, R\sin(\phi)\sin(\theta), \, R\cos(\phi))$. In this case, the magnitude of the normal vector would be $$ \| n \| = R^2 \sin(\phi) $$
Integrate this over the domain $0 \leq \theta \leq 2\pi$ and $0 \leq \phi \leq \pi$ to cover the whole sphere, and it gives the desired answer.