A rigid sphere moves on a trajectory, made up of points that come out of a mechanical simulation.
To a visual inspection, the trajectory is helical. I calculate the axis $\boldsymbol{N}$ by using the average of tangent vectors between an integer number of turns of the helix. I then estimate the radius $R$ by projecting each of the points onto the plane perpendicular to $\boldsymbol{N}$, and fitting them to a circle. Therefore I know the approximate axis $\boldsymbol{N}$ and radius $R$ of the helix.
I want to be able to check whether a certain section of the sphere is constantly facing the axis of the helix, or away from it. By "a section" I mean a certain circular patch on the surface of the helix.
If I define this patch $S$, I can calculate the normal at its centre $\boldsymbol{r}_0$ as $\boldsymbol{n}_S(r_0)$. What's the best quantity to study to investigate the relative orientation between the sphere and helix axis $\boldsymbol{k}$?
The dot product between $\boldsymbol{k}$ and as $\boldsymbol{n}_S(r_0(t))$?
Suppose that $\alpha\colon I\subset \mathbb{R}\to\mathbb{R}^3$ is the parametrization of of the helicoidal trajectory, i.e. the position of the centre of the sphere in function of the time. Call the helix's radius $\rho$. Then for every $t$ the point $$ \alpha(t) + \rho N(t) $$ lies the on the axis of the helix. Here $N(t)$ is the normal vector field along $\alpha$. The direction of $N(t)$ is given by $$ \alpha''(t) - \left\langle \alpha''(t), \frac{\alpha'(t)}{\|\alpha'(t)\| }\right\rangle \frac{\alpha'(t)}{\|\alpha'(t)\|}. $$ If you want to get the unit vector $N(t)$, you still need to divide it by its length. In mechanical terms, $N(t)$ points in the direction of the normal acceleration of the curve $\alpha$, hence we subtract the tangent acceleration from $\alpha''(t)$. If the speed of $\alpha$ is constant, then the formula for $N(t)$ is easier: $$ N(t) = \frac{\alpha''(t)}{\|\alpha''(t)\|}. $$ So for a small patch $S$ a good quantity might be $$ \langle n_S(r_0(t)), N(t) \rangle. $$ The closer this quantity lies to one, the more the patch faces the axis.
Update: If you don't know the trajectory, but do know the axis, you can find $N(t)$ as you suggested in the comments. Say $k$ is the unit direction of the axis and $q$ is any point on the axis. At a point $p$ approximately on the helix, the vector $$ (p-q) - \langle (p-q), k \rangle. $$ points to the axis. The length of this vector should be approximately $R$.