In Section 5.3 of Humphreys book on Coxeter/Reflection groups, he develops a geometric representation of a Coxeter system $(W,S)$ by taking an $\mathbb{R}$-vector spaces $V$ with basis $\{\alpha_s:s\in S\}$, and defining a symmetric bilinear form $B$ on $V$ by $$ B(\alpha_s,\alpha_{s'})=-\cos\frac{\pi}{m(s,s')} $$ where $m(s,s')$ is the order of $ss'$ in $W$. For $s\in S$, he defines a reflection $$ \sigma_s\lambda=\lambda-2B(\alpha_s,\lambda)\alpha_s $$
In $I_2(4)$, $m(s,s')=4$, hence $$ \sigma_s(\alpha_{s'})=\alpha_{s'}-2B(\alpha_s,\alpha_{s'})\alpha_s=\alpha_{s'}+2\cos\frac{\pi}{4}\alpha_s=\alpha_{s'}+\sqrt{2}\alpha_s $$ and in $I_2(6)$ where $m(s,s')=6$, $$ \sigma_s(\alpha_{s'})=\alpha_{s'}-2B(\alpha_s,\alpha_{s'})\alpha_s=\alpha_{s'}+2\cos\frac{\pi}{6}\alpha_s=\alpha_{s'}+\sqrt{3}\alpha_s. $$
I'm confused because I thought the action of the Coxeter group should send roots to roots, and in these cases aren't the roots supposed to be integral linear combinations of simple roots?
By the strictest definition, $\alpha_s$ and $\alpha_{s'}$ are not roots, just ordinary vectors used to define this representation, so there's no contradiction when you get coefficients that are not integers.
However, they do behave just like simple roots if you drop the integer coefficient condition, and you will get exactly $8$ for $I_2(4)$ and $12$ for $I_2(6)$.