Why is the dimensionality of the $SL(N,{\mathbb{R}})$ and $SU(N)$ groups $N^2-1$ but only $N^2$ for $GL(N,{\mathbb{R}})$ and $U(N)$?

Question

As far as I know the only difference between the two special and normal groups is that the $SL(N,{\mathbb{R}})$ and $SU(N)$ generators must be traceless.

I understand that the dimensionality corresponds to the number of linearly independent matrices in the algebra. So how does the tracelessness of the generating matrices affect their linearity?