Motivation for definition of bilinear form in linear representation of Coxeter groups?

Question

In a set of notes on Coxeter groups I am reading the following definitions are made:

1. Let $M = (m_{ij})_{1 \leq i,j \leq n}$ be a symmetric $n \times n$ matrix with entries from $\mathbb{N} \cup \{ \infty \}$ such that $m_{ii} = 1$ for all $1 \leq i \leq n$ and $m_{ij} > 1$ whenever $i \neq j.$ The Coxeter group of type $M$ is the group $$W (M) = \langle \{s_1, \dots, s_n \} \mid \{s_is_j)^{m_{ij}} = 1 \mid 1\leq i , j \leq n, m_{ij} < \infty \} \rangle.$$

2. Let $V$ be a real vector space with basis $(e_i)_{i \in [n]}.$ Denote by $B_M$ the symmetric bilinear form on $V$ determined by $$B(e_i, e_j) = -2\cos (\pi/m_{ij})$$ for $i,j \in [n]$ and such that $B(e_i, e_j) = -2$ if $m_{ij} = \infty$.

3. Define the linear transformations $\rho_i (i \in [n]) $ by $\rho_i (x) = x - B(x, e_i)e_i \quad (x \in V)$

The linear representation of the Coxeter group of type $M$ is then given by the homomorphism $w \to \rho_w.$ I now have two questions:

1. Why the choice of the this bilinear form?

2. If $V = \mathbb{R}^n$, is this notion of a reflection consistent with reflections formed by fixing a hyperplane under the usual inner product?

Answer 1

The bilinear form is chosen this way to meet the constraint $(\rho_i\rho_j)^{m_{ij}}=1$. For example with $$ M=\left(\begin{array}{cc}1&3\\3&1\end{array}\right) $$ we get that $-2\cos(\pi/3)=-1$, and therefore $$ \rho_1:\begin{cases}e_1\mapsto-e_1\\e_2\mapsto e_1+e_2\end{cases} $$ $$ \rho_2:\begin{cases}e_1\mapsto e_1+e_2\\e_2\mapsto -e_2\end{cases} $$ when $$ \rho_1\rho_2:\begin{cases}e_1\mapsto e_2\\e_2\mapsto -e_1-e_2\end{cases} $$ and we can easily verify that $(\rho_1\rho_2)^3$ is the identity transformation.

In general the transformations $\rho_i$ are not always reflections w.r.t. a hyperplane of $\Bbb{R}^n$. Consider the case $$ M=\left(\begin{array}{cc}1&\infty\\\infty&1\end{array}\right). $$ In this case the matrix $B(e_i,e_j)$ is $\pmatrix{2&-2\cr-2&2\cr}$ singular, and we easily see that $e_1+e_2$ is fixed by both $\rho_1$ and $\rho_2$. So if both $\rho_1$ and $\rho_2$ where ordinary reflections, they would be equal transformations of the plane. But clearly $\rho_1\neq\rho_2$, so this is not the case.

If $W(M)$ is finite and $B$ is positive definite (in particular non-degenerate), then we can turn $B$ into an invariant inner product by the usual averaging argument. The relation $\rho_i(e_i)=-e_i$ then forces $\rho_i$s to be reflections. I don't know about the other cases (i.e. whether this sufficient condition is necessary). IIRC if $W(M)$ is a finite group, then $B$ is automatically positive definite.