In a Coxeter group acting on a vector space $V$, if $\alpha$ is a simple root, there is a simple reflection $s_\alpha$ defined on $V$ by $$ s_\alpha(\lambda)=\lambda-2B(\alpha_s,\lambda)\alpha_s $$ where $B(\alpha_s,\alpha_{s'})=-\cos\frac{\pi}{m(s,s')}$, with $m(s,s')$ the order of $ss'$ in the Coxeter group.
When I try to use this formula for a system of type $B_n$, something isn't working. Suppose $\alpha$ and $\beta$ are the simple roots corresponding to the end of the Dynkin diagram, with edge labelled $4$ connecting their corresponding vertices. Then $$ s_\beta(\alpha)=\alpha-2B(\alpha,\beta)\beta=\alpha+2\cos\frac{\pi}{4}\beta=\alpha+\sqrt{2}\beta. $$
This should be a root, but roots in a crystallographic system only have integral coefficients. where have I applied the formula incorrectly?
Something is wrong with your formula; there should be an extra "scaling" factor, it should eventually be
$s_a(\lambda) = \lambda + 2 \frac{||\lambda||}{||\alpha||} \cos(\frac{\pi}{m(s_\lambda, s_\alpha)}) \alpha$.
Accordingly, the definition of what you call $B(\cdot, \cdot)$ does not make it a bilinear form; it would give all roots the same length, which is exactly not the case in the considered example, where $||\alpha|| = \sqrt{2} \cdot ||\beta||$ and thus
$s_\beta(\alpha) = \alpha + 2\beta$.