I'm puzzled about the $"*"$ in the following notation for Lie groups: $SO^* (2N)$ or $SU^* (2N)$.
I don't understand what is the meaning of this notation. It is introduced for example in Gilmore (Lie Groups, Lie Algebras and some of their applications) on pag. 47. It' s probably related to sesquilinear metrics, but I don't understand how.
The star means nothing in itself, it's like the prime (') sign, it means $\mathrm{SO}^*(n)$ something analogous but different from $\mathrm{SO}(n)$. Actually, $\mathbf{H}$ denoting the quaternions, $\mathrm{SO}^*(n)$ (or $\mathrm{SO}^*(2n)$, I'm not sure of conventions) is the group of $\mathbf{H}$-linear automorphisms of $\mathbf{H}^n$ preserving a nondegenerate "(7/8)-bilinear" form, namely an hermitian form with respect to a the anti-involution $a+bi+cj+dk\mapsto a+bi-cj+dk$ of $\mathbf{H}$ (which is not the standard anti-involution!). I'm more used to the notation $\mathrm{SO}(n,\mathbf{H})$.
The notation $\mathrm{SU}^*$ refers to "(5/8)-bilinear" forms, namely with respect to the standard anti-involution of $\mathbf{H}$; unlike the previous ones these one have a natural signature as usual real orthogonal form or complex hermitian forms.
See Witte-Morris's book for an introduction to classical Lie groups (see p415 in the version 5 from arxiv).