What is the outward-pointing normal to the boundary of the spherical cap in the figure below at the point $p$ ? We are given that $v$ is the vertex (pole ..not sure what to call it) of the cap and I can compute the tangent of the curve $C$ that connects $v$ to $p$. The curve $C$ is a piece of a great circle from the pole $v$. I believe this tangent is the normal to the boundary at $p$. Correct ?
If the "cap" consists only of the spherical surface $S$ the point $p$ is a boundary point of $S$. You can still consider the line $O\vee p$ connecting the center $O$ of the sphere with $p$ as normal line to the surface at $p$. When you have a parametric representation $(u,v)\mapsto {\bf r}(u,v)$ of $S$, and $p={\bf r}(u_0,v_0)$, then the outward normal ${\bf n}$ at $p$ is one of $$\pm\>{\bf r}_u\times {\bf r}_v\biggr|_{(u_0,v_0)}\ .$$ The selection of the sign has to be made at the end of the computation. This vector ${\bf n}$ is orthogonal as well to the tangent to $c$ at $p$ as to the latitude circle bounding $S$, because both these curves are lying in $S$. (Note that a surface in ${\mathbb R}^3$ has $\pm 1$ normal directions at each point $p\in S$, but a curve $\gamma$ in ${\mathbb R}^3$ has at each point $p\in\gamma$ a full normal plane with infinitely many directions.)
If, however, the "cap" $C$ is considered as a three-dimensional body, bounded by the spherical surface $S$ and a horizontal disc $D$, then the boundary $\partial C$ has no normal at $p$. For purposes of integration (e.g., when computing the flux of some vector field through $\partial C$) this is of no importance.