I'm trying to follow Example 9 in Section 1.5 of Macdonald's book "Symmetric Functions and Hall Polynomials". I have trouble with understanding some points.
Before stating my question, I will first explain the example briefly. If $R$ is a root system, then the Weyl denominator formula for $R$ is $$\sum_{w\in W}\varepsilon(w)e^{w\rho}=\prod_{\alpha\in R^{+}}(e^{\alpha/2}-e^{-\alpha/2}),$$ where $W$ is the Weyl group of $R$, $R^{+}$ is the set of positive roots (relative to some base), $\rho$ is the half-sum of the positive roots and $\varepsilon(w)$ is the sign of $w\in W$. The goal of the example is to express the products $$\prod_{i<j} (1-x_{i}x_{j}),\quad\prod_{i}(1-x_{i})\prod_{i<j}(1-x_{i}x_{j}),\quad\prod_{i}(1-x_{i}^{2})\prod_{i<j}(1-x_{i}x_{j})$$ as linear combinations of Schur functions using the Weyl denominator formula for root systems of types $D_{n}, B_{n}, C_{n}$, respectively. More precisely, \begin{align*} \prod_{i<j}(1-x_{i}x_{j})&=\sum_{\pi}(-1)^{|\pi|/2}s_{\pi}(x_{1},\ldots,x_{n})\\ \prod_{i}(1-x_{i}^{2})\prod_{i<j}(1-x_{i}x_{j})&=\sum_{\mu}(-1)^{|\mu|/2}s_{\mu}(x_{1},\ldots,x_{n})\\ \prod_{i}(1-x_{i})\prod_{i<j}(1-x_{i}x_{j})&=\sum_{\nu}(-1)^{(|\nu|+p(\nu))/2}s_{\nu}(x_{1},\ldots,x_{n}), \end{align*} where the first sum ranges over all partitions $\pi=(\alpha_{1}-1,\ldots,\alpha_{p}-1\mid\alpha_{1},\ldots,\alpha_{p})$ with $\alpha_{1}\leq n-1$, the second sum ranges over all partitions $\mu=(\alpha_{1}+1,\ldots,\alpha_{p}+1\mid\alpha_{1},\ldots,\alpha_{p})$ with $\alpha_{1}\leq n-1$, the third sum ranges over all self-conjugate partitions $\nu=(\alpha_{1},\ldots,\alpha_{p}\mid\alpha_{1},\ldots,\alpha_{p})$ with $\alpha_{1}\leq n-1$ and $p(\nu)=p$.
Now I'm ready to state my question. Here I will just consider the first product. Let $\epsilon_{1},\ldots,\epsilon_{n}$ be the standard basis for $\mathbb{R}^{n}$. The root system $R=D_{n}$ is given by $$\{\pm(\epsilon_{i}\pm\epsilon_{j}):i\neq j\},$$ and the subset $$\Delta=\{\epsilon_{1}-\epsilon_{2},\epsilon_{2}-\epsilon_{3},\ldots,\epsilon_{n-1}-\epsilon_{n},\epsilon_{n-1}+\epsilon_{n}\}$$ is a base. Then \begin{align*} R^{+}&=\{\epsilon_{i}\pm\epsilon_{j}:1\leq i<j\leq n\},\\ \rho&=(n-1)\epsilon_{1}+(n-2)\epsilon_{2}+\cdots+\epsilon_{n-1}. \end{align*} On the other hand, the reflection $\sigma_{\epsilon_{i}-\epsilon_{i+1}}$ in the orthogonal complement of $\epsilon_{i}-\epsilon_{i+1}$ switches $\epsilon_{i}$ with $\epsilon_{i+1}$, and $\sigma_{\epsilon_{i}+\epsilon_{i+1}}$ switches $\epsilon_{i}$ to $-\epsilon_{j}$ and $\epsilon_{j}$ to $-\epsilon_{i}$. Thus the Weyl group $W$ can be identified with the group of all elements that permute $n$ elements as well as switching an even number of their signs, hence isomorphic to $S_{n}\ltimes\mathbb{Z}_{2}^{n-1}$. If we replace $e^{-\epsilon_{i}}$ by $x_{i}$, the right-hand side of the denominator formula becomes: \begin{align*} \prod_{\alpha\in R^{+}}(e^{\alpha/2}-e^{-\alpha/2})&=e^{\rho}\prod_{\alpha\in R^{+}}(1-e^{-\alpha})\\ &=x_{1}^{-(n-1)}x_{2}^{-(n-2)}\cdots x_{n-1}^{-1}\prod_{i<j}(1-x_{i}x_{j})\left(1-\frac{x_{i}}{x_{j}}\right)\\ &=(x_{1}\cdots x_{n})^{-(n-1)}\prod_{i<j}(1-x_{i}x_{j})\left(x_{j}-x_{i}\right). \end{align*} The product $\prod_{i<j}(1-x_{i}x_{j})$ is the product we are interested in and the product $\prod_{i<j}(x_{j}-x_{i})$ is involved in the definition of Schur function. So it remains to relate the left-hand side of the denominator formula to the partitions $\pi=(\alpha_{1}-1,\ldots,\alpha_{p}-1\mid\alpha_{1},\ldots,\alpha_{p})$ with $\alpha_{1}\leq n-1$. But I have no idea. If someone has any idea, please let me know. Thank you for reading.
First, we would like to fix some notations. Instead of $\varepsilon_i$ (as in the OP's notation), we use $v_i$ (as in Macdonald) for the standard basis for $\mathbb R^n$.
We solve this problem in several steps:
(Step 1) Rewrite the equation into a sum over $\mathbb Z_2^{n-1}$;
(Step 2) Rewrite the summand as Schur functions indexed by a partition $\lambda(\epsilon)$ depending on $\epsilon\in\mathbb Z^2_{n-1}$;
(Step 3) Prove that $\epsilon\mapsto \lambda(\epsilon)$ gives a bijection between $\mathbb Z_2^{n-1}$ and the desired set of partitions;
(Step 4) Check the signs.
(Step 1) Identify $\mathbb Z_2^{n-1}$ as $\epsilon=(\epsilon_1,\dots,\epsilon_n)\in\mathbb Z_2^n$, such that $\prod_i \epsilon_i=1$. Let $w\in S_n$, then $w\epsilon :v_i\mapsto\epsilon_{i} v_{w(i)}$, and \begin{align} w\epsilon\rho=w\sum \epsilon_i(n-i)v_i = \sum \epsilon_i(n-i)v_{w(i)}, \end{align} where $\rho=\sum (n-i)v_i$.
Under the identification $e^{-v_i}=x_i$, we now need to show that \begin{align}\tag{*} \frac1{a_\delta} \sum_{\epsilon\in\mathbb Z_2^{n-1}} \varepsilon(\epsilon) \sum_{w\in S_n} \varepsilon(w) \prod_i x_{w(i)}^{n-1-\epsilon_i(n-i)} \end{align} can be written as a sum over Schur functions.
(Step 2) Fix $\epsilon\in\mathbb Z_2^{n-1}$, define $\overline\lambda(\epsilon)=\delta-\mathbf1-\epsilon(n\mathbf1-\delta)\in\mathbb Z^n$, i.e., \begin{align} \overline\lambda(\epsilon)_i:=i-1-\epsilon_i(n-i), \quad 1\leq i\leq n, \end{align} and define $\lambda(\epsilon)$ by \begin{align} \lambda(\epsilon) = (\overline\lambda(\epsilon)+\delta)^+-\delta, \end{align} where $\delta = (n-1,\dots,1,0)$ and $(\cdot)^+$ is sorting in descending order.
Then \begin{align*} \mathrel{\phantom{=}} \sum_{w\in S_n} \varepsilon(w) \prod_i x_{w(i)}^{n-1-\epsilon_i(n-i)} &= \sum_{w\in S_n} \varepsilon(w) \prod_i x_{w(i)}^{\overline\lambda(\epsilon)_i+n-i} \\ &= \sum_{w\in S_n} \varepsilon(w) w(x^{\overline\lambda(\epsilon)+\delta}) = a_{\overline\lambda(\epsilon)+\delta} = s(\epsilon) a_{\lambda(\epsilon)+\delta}, \end{align*} where $s(\epsilon)$ is some sign.
Now (*) is equal to $$\sum_{\epsilon\in\mathbb Z_2^{n-1}} \varepsilon(\epsilon)s(\epsilon) s_{\lambda(\epsilon)}.$$
(Step 3) Fix $\epsilon\in\mathbb Z_2^{n-1}$. Let $I=I(\epsilon)=\{i_1<\dots<i_l\}$ denote the indices with negative sign in $\epsilon$ and $J=I^c=\{j_1<\dots<j_{n-l}\}$ its compliment.
By the definition of $\overline\lambda(\epsilon)$, we have \begin{align} (\overline\lambda(\epsilon)+\delta)_i = n-1-\epsilon_i(n-i) = \begin{cases} 2n-1-i,&\epsilon_i=- \\ i-1,&\epsilon_i=+ \\ \end{cases}. \end{align} From the formula above, the index of the entry of $\overline\lambda(\epsilon)+\delta$ in descending order is given by $i_1,\dots,i_l,j_{n-l},\dots,j_1$ (i.e., the $i_1$th entry is the greatest, then the $i_2$th entry, and so on...). Hence $\lambda(\epsilon)$ is given by \begin{align} \lambda(\epsilon)_m = \begin{cases} n+m-1-i_m, & 1\leq m\leq l; \\ j_{n+1-m}-n-1+m, & l+1\leq m\leq n, \end{cases} \end{align} which is indeed a partition and the sign $s(\epsilon)$ is given by moving the $i_k$th entry to the $k$th entry, i.e., $s(\epsilon)=(-1)^{\sum_{k=1}^l (i_k-k)}$.
Define \begin{align} \alpha(\epsilon) = \begin{cases} (n-i_k)_{1\leq k\leq l-1}, &\epsilon_n=-; \\ (n-i_k)_{1\leq k\leq l}, &\epsilon_n=+. \\ \end{cases} \end{align} Let $p=l-1$ or $l$ be the length of $\alpha$ depending on $\epsilon$. It is clear that $\alpha$ is strictly decreasing and each entry in $\alpha$ is at least 1 and that $\alpha_1\leq n-1$.
We will show that the Frobenius notation for $\lambda(\epsilon)$ is exactly $\left(\alpha(\epsilon)-\mathbf1\mid\alpha(\epsilon)\right)$.
Firstly, we claim that the maximal square in $\lambda(\epsilon)$ has length $p$. In fact, the length is equal to $\#\{m:\lambda(\epsilon)_m-m\geq0\}$. Note the following: when $m\leq l-1$, $\lambda(\epsilon)_{l-1}\geq0$; when $m=l$, $\lambda(\epsilon)_m-m = n-1-i_m\geq0$ if $\varepsilon_n=+$ and $\lambda(\epsilon)_m-m<0$ if $\varepsilon_n=-$. when $m\geq l+1$, $\lambda(\epsilon)_m-m<0$.
Secondly, note that the first $p$ rows of $\left(\alpha(\epsilon)-\mathbf1\mid\alpha(\epsilon)\right)$ are equal to those of $\lambda(\epsilon)$ by definition.
Lastly, we need to show that the first $p$ columns of them are also equal. For $1\leq k\leq p$, the $k$th column of $\left(\alpha(\epsilon)-\mathbf1\mid\alpha(\epsilon)\right)$ is $\alpha_k+k=n-i_k+k\geq l+1$, while the $k$th column for $\lambda(\epsilon)$ is $\#\{i: \lambda_i\geq k\}$.
To show that $n-i_k+k \leq \#\{i: \lambda_i\geq k\}$, we need $\lambda_{n-i_k+k}\geq k$, i.e., $$ j_{i_k-k+1}\geq i_k+1. $$ This holds because $i_k-k$ is the number of +'s in $[1,i_k]$, and $j_{i_k-k+1}$ is the index of the $(i_k-k+1)$th $-$ sign which is at least $i_k+1$.
Similarly, to show that $1+n-i_k+k > \#\{i: \lambda_i\geq k\}$, we need $\lambda_{1+n-i_k+k} < k$, i.e., $$ j_{i_k-k}< i_k. $$ This holds because $i_k-k$ is the number of +'s in $[1,i_k]$, and $j_{i_k-k}$ is the index of the $(i_k-k)$th $+$ sign which is strictly less than $i_k$. This concludes the proof that $\lambda(\epsilon)=\left(\alpha(\epsilon)-\mathbf1\mid\alpha(\epsilon)\right)$.
Note that the following bijections: \begin{align*} \mathbb Z_2^{n-1} &\leftrightarrow \{I(\epsilon)\subset[1,n]: \#I(\epsilon)\text{ is even}\} \\ &\leftrightarrow \{I(\epsilon)\setminus\{n\}\subset[1,n-1]\}\\ &\leftrightarrow \{\alpha(\epsilon): \alpha_1>\dots>\alpha_p\geq1, \alpha_1\geq n-1\}. \end{align*}
(Step 4) Now, it suffices to show that \begin{align} \varepsilon(\epsilon) s(\epsilon) = (-1)^{|\lambda(\epsilon)|/2}. \end{align} Note that $$|\lambda(\epsilon)|/2 = |\alpha(\epsilon)| = \sum_{k=1}^p (n-i_k) = \sum_{k=1}^l (n-i_k)=\sum_{k=1}^l (k-i_k)+\sum_{k=1}^l (n-k)$$ since when $\epsilon_n=+$, $p=l$; when $\epsilon_n=-$, $p=l-1$ and the $l$th term is $n-i_l=0$. Also, we have $\varepsilon(\epsilon)s(\epsilon) = (-1)^{l/2+\sum_{k=1}^l (i_k-k)}$. Now, since $l$ is even, we have $$\sum_{k=1}^l (k-i_k)+\sum_{k=1}^l (n-k) \equiv \sum_{k=1}^l (k-i_k)+\binom{l+1}{2} \equiv \sum_{k=1}^l (k-i_k)+\frac{l}{2}.$$