The textbook answer is $\frac{180!} {(5!)^{36}36!}$. This is because the total permutations of 180 items is 180!. The 180! gets divided by $(5!)^{36}$ because the order in each group of 5 doesn't matter. It then gets divided by 36! because the order of the 36 groups doesn't matter.
I calculated this a different way from the textbook. First, I counted groups of 5 from 180 items, which is ${180 \choose 5}=\frac{180!}{5!175!}$ ways. Now from all these different groups of 5, I counted groups of 36, which is ${{180 \choose 5} \choose 36}=\frac{\frac{180!}{5!175!}!}{36!(\frac{180!}{5!175!}-36)!}$.
Is the way that I counted (ie, first count groups of 5, then among those groups, count groups of 36) a valid way to solve this problem? The answer I got was so different from the textbook's I'm not sure how to verify it.
Yeah, your way of counting doesn't work, for the reason that there will be significant overlaps between the groups counted by $\binom{180}{5}$. If your set is $\lbrace 1, \ldots, 180 \rbrace$, your counting allows for $\lbrace 1, 2, 3, 4, 5\rbrace$ and $\lbrace 1, 2, 3, 4, 6\rbrace$ to be two of your groups.
To follow your reasoning to the answer given in the book, you can count $\binom{180}{5}$ for your first group, then with these $5$ removed (whatever they are), you can count $\binom{175}{5}$ for your second group. Removing these $5$ more ($10$ in total), you can choose one of $\binom{170}{5}$ for your third group, etc, giving you the following count of ordered lists of unordered groups:$\require{cancel}$
\begin{align*} \binom{180}{5} \cdot \binom{175}{5} \cdot \ldots \cdot \binom{10}{5} \cdot \binom{5}{5} &= \frac{180!}{\cancel{175!} \cdot 5!} \cdot \frac{\cancel{175!}}{\cancel{170!} \cdot 5!} \cdot \ldots \cdot \frac{\cancel{10!}}{\cancel{5!} \cdot 5!} \cdot \frac{\cancel{5!}}{0! \cdot 5!} \\ &= \frac{180!}{(5!)^{36}}. \end{align*} However, since we want an unordered group of groups, we have counted each unordered group $36!$ times, yielding a result of $$\frac{180!}{(5!)^{36} \cdot 36!}.$$