Chapter 8 section 5 of Linear Algebra and Its Applications by David C. Lay defines a polytope in $\mathbb R^n$ as the convex hull of a finite set of points. He then says that a polygon is just a $2$-polytope.
Doesn't this definition of polygons exclude all concave polygons? If so, why is this the definition he uses since it isn't consistent with other definitions of polygons?
Yes, he is defining convex polytopes but is not saying so explicitly. I don't know if he does make it explicit elsewhere.
It is a common slip of rigour in certain areas of mathematics. I quite often find people quoting from, say, Grünbaum's Convex Polytopes and asserting angrily that the quote omits the word "convex" so it must be correct to do so. When I point out the he expressly states at the beginning that he is discussing convex polytopes but will in general omit the word because its repetition would be tedious, and that other authors should also make their omission explicit, all suddenly goes quiet.
In Regular Polytopes, Coxeter brashly suggested that generalizing such definitions to star and other non-convex polytopes is straight forward. In fact it is fraught with inconsistencies, and in disciplines such as topology or abstract polytope theory one must resort to quite different approaches to definition.