Suppose I start with two points $p_1, p_2 \in \mathbb{R}^3$. I want to interpolate along a circular arc between these two points, given normal vectors $n_1, n_2$ at each point.
It's fairly straightforward to compute a radius and center for the circle when there is a reasonable amount of curvature. I can compute the angle between the normals as $\theta = \cos^{-1}(n_1 \cdot n_2)$, then $r = \frac{\lvert p_2 - p_1 \rvert}{2\sin\frac{\theta}{2}}$. From this, it's fairly easy to compute the interpolated points.
However, as the curvature $\kappa$ approaches 0, evaluating these equations becomes problematic. In the limit, the curve flattens to a straight line between the points and $r \to \infty$.
If the expectation is that curvature may by small (or even exactly zero), is there a more numerically stable formulation of the equation of the arc to allow computation of the interpolated points?
I've considered just setting a cutoff on the curvature and interpolating using a line if it gets too small, but this feels kludgey and I'd probably be unnecessarily sacrificing some accuracy.
Suppose the two given points are $\mathbf{P}_0$ and $\mathbf{P}_1$, and the normals are $\mathbf{N}_0$ and $\mathbf{N}_1$. Using the normals, you can construct tangent lines at $\mathbf{P}_0$ and $\mathbf{P}_1$, and intersect these to get a third point $\mathbf{P}_a$ (here "$a$" stands for "apex"). The three points $\mathbf{P}_0$, $\mathbf{P}_a$, $\mathbf{P}_1$ define a quadratic Bezier curve (i.e. a parabola) that satisfies your constraints.
If you want some more flexibility, and you want to be able to reproduce circles (in the symmetric cases where this is possible), then you need to use rational quadratics, rather than polynomial ones. A good choice of weights for your rational quadratic is $w_0 = w_1 = 1$ and $w_a = \cos \tfrac12\theta$, where $\theta$ is the angle between the two normals. If the normals make the same angle with the chord $\mathbf{P}_0\mathbf{P}_1$, then this will give you a circular arc. If not, you'll still get a "nice" elliptical curve. In case you're not familiar with rational Bezier curves, the equation of the curve is $$ \mathbf{P}(t) = \frac{ (1-t)^2\mathbf{P}_0 + 2t(1-t)w_a\mathbf{P}_a + t^2 \mathbf{P}_1 }{ (1-t)^2 + 2t(1-t)w_a + t^2 } $$