$g(x)= x+3$, therefore $g^{-1}(x) = x-3$
$h(x)= 2x-4$, therefore $h^{-1}(x) = (x+4)/2$
I thought of $g^{-1} \circ h^{-1}$ as $(g^{-1} \circ h^{-1})(x)$ and got $0.5x-1$, but the correct answer (from the book) is $0.5x-3$
What is the difference in the notation and how to use it?
There isn't anything different about the notation $$ (g^{-1}\circ h^{-1})(x) = g^{-1}(h^{-1}(x)) = g^{-1}\left(\frac x2 + 2\right)= \frac x2+2 - 3 = \frac x2 -1. $$ Likewise $$ (h^{-1}\circ g^{-1})(x) = h^{-1}(g^{-1}(x)) = h^{-1}(x-3)= \frac{(x-3)}{2} +2 = \frac x2 +\frac 12. $$