Let $X$ be a complete toric variety of dimension $n$. It is a classical result that if $D$ is a toric nef divisor, then its degree $D^n$ can be computed as the Volume of the corresponding polytope $P_D\subset M_{\mathbb{R}}$. My question is what kind of volume is this? Lattice volume? Volume with respect to Haar measure giving the lattice $M$ covolume $1$? Are these the same?
Also, does this mean that if I start with a nef toric divisor which is not big (this is equivalent to the polytope not being full-dimensional), then it's degree is $0$? or would it be equal to some kind of relative volume?
For the first question: it's just the usual volume on $\mathbf R^n$, but normalised so that (if I remember correctly) a fundamental region of the lattice has volume 1. (You should check that.)
Yes, a nef divisor which is not big must have top selfintersection $D^n=0$, or equivalently zero volume. This has nothing to do with toric geometry: it's true for any line bundle on an irreducible projective variety. See Theorem 2.2.16 of Lazarsfeld Volume 1.