In Hatcher, the notation $X/A$ as appearing in the following text is never defined:
If $(X,A)$ is a CW Pair consisting of a cell complex $X$ and a subcomplex $A$, then the quotient space $X/A$ inherits a natural cell complex structure from $X$ [...]
My guess as to what this notation means is to identify everything in $A$ to a single point. This is just a guess though, is it correct?
Yes, $X/A$ is the space where you identify everything in $A$ to a single point, also known as the quotient space of $X$ by $A$.