In 5.4 of his book Lectures on Invariant Theory, Igor Dolgachev introduces the dual of a root by requiring that
- $\check\alpha(t) f_\alpha(x) \check\alpha^{-1}(t)= f_\alpha(x)$
- $(\alpha\circ\check\alpha)(t)=t^2$, i.e. $\langle\alpha,\check\alpha\rangle=2$.
He earlier defined a root to be a multiplicative character $\alpha:T\to\mathbb{G}_{\mathrm{m}}$ of the torus such that there exists an additive one-parameter subgroup $f_\alpha:\mathbb{G}_{\mathrm{a}}\to G$ with $$ \forall t\in T: \forall x\in\Bbbk: t\cdot f_\alpha(x)\cdot t^{-1} = f_\alpha(\alpha(t)x).$$ Now by the above and (2), we have to have $$\check\alpha(t) f_\alpha(x) \check\alpha^{-1}(t)= f_\alpha((\alpha\circ\check\alpha)(t) x) = f_\alpha(t^2 x).$$ which is a contradiction to (1), at least in general. Now, I am worried that he means something else by (1) because $\check\alpha$ is not uniquely defined when requiring only (2). In fact, in the example given, one could define either \begin{align*} \check\alpha(t) &:= I_r + (t-1)E_{ii} + (t^{-1}-1)E_{ii} & \text{or}\\ \check\alpha(t) &:= I_r + (t^2-1)E_{ii} - E_{ii} \end{align*} and both would satisfy (2).
The question is, which additional condition is needed to make this definition of a dual root unambiguous?
Edit Maybe I should have said this right away: I would expect that the additional condition should be that $\beta-\langle\check\alpha,\beta\rangle\alpha$ is a root for all roots $\beta$. This would be coherent with the corresponding definition of abstract root systems. However, I am not sure if I am missing something.