I am having difficulty understanding how to interpret the coordinate system proposed by Spencer. From his handbook Field Theory Handbooks ISBN 9783540184300 proposed a system with the domains illustrated in the picture.
The domain of eta represents the curvilinear axis represented by the ellipse rotating around the minor axis. What exactly is the characteristic of the ellipse in the picture? Is 'Eta' representing a number describing the minor or mayor axis of the ellipse?

Neither or both, depending on your point of view. Refer to the illustration: the surfaces with $\eta$ held constant are oblate spheriods. From the equations of the transformation to Cartesian coordinates, these spheroids have the parameterization $$\begin{align}x &= (a\cosh\eta)\sin\theta\cos\psi \\ y &= (a\cosh\eta)\sin\theta\sin\psi \\ z &= (a\sinh\eta)\cos\theta.\end{align}$$ I’ve added parentheses to emphasize the constant quantity in each equation. The half-axis lengths of the spheroid are therefore $a\cosh\eta$ and $a\sinh\eta$. Similarly, the surfaces with constant $\theta$ are hyperboloids of revolution with half-axis lengths $a\cos\theta$ and $a\sin\theta$. The angle $\theta$ represents the aperture half-angle of the hyperboloid’s asymptotic cone. Surfaces with constant $\psi$ are half-planes that make an angle of $\psi$ with the positive $x$-$z$ half-plane.