The book I'm reading says the quotient of a semi-simple group by its maximal triangularizable subgroup can fail to have finite volume. For the case of $SL(n,\mathbb{R})$ I'm not really seeing it. In the KAN decomposition the maximal triangular subgroup is AN, and compact subgroup is $K\simeq SO(n)$ which has finite volume.
Is there any intuitive way of understanding when do such situations happen?
The title is somewhat misleading: it is any easy exercise to show every compact Lie group has finite (Haar) volume.
You are correct that for the Iwasawa decomposition $G=KAN$ the quotient $G/AN$ is diffeomorphic to $K$; the issue is that $K$ need not be compact. $K$ is the maximal compact subgroup if the center of $G$ is finite, but without this requirement it may not be.
As a counterexample, take $G=\widetilde{\operatorname{SL}(2,\mathbb{R})}$ to be the universal cover of $\operatorname{SL}(2,\mathbb{R})$, a connected semisimple group for which $K\cong\mathbb{R}$.