I want to convert a 2nd degree rational Bézier curve in 3 dimensions (4 if in homogeneous coordinates) into a conic parametric curve.
Given a rational bezier curve of degree 2: $$B(t) = \frac{(1-t)^2 P_0 w_0 + 2 t (1 - t) P_1 w_1 + t^2 P_2 w_2}{(1-t)^2 w_0 + 2 t (1 - t) w_1 + t^2 w_2}, P_i \in \mathbb{R}^3, w_i \in \mathbb{R} $$
I want to convert this to an equal curve of the form
$E(s) = C_E + f_1 cos(s) + f_2 sin(s)$ if it is an ellipse, or
$H(s) = C_H + f_1 cosh(s) + f_2 sinh(s) $ if it is a hyperbola
Alternative form: $B(t) = \frac{t^2A+tB+C}{t^2w_A+tw_B+w_C}$, with
$A = P_0 w_0 - 2 P_1 w_1 + P_2 w_2$
$B = 2 P_1 w_1 - 2 P_0 w_0$
$C = P_0 w_0$
I have a partial solution for the ellipse case:
I'm not exactly sure why, but it seems that at the point $t_0$ where the weight term $t^2w_A+tw_B+w_C$ has its extrema ($2 t_0 w_A + w_B = 0 \Rightarrow t_0 = -w_B / 2 / w_A$), the segment from $B(t_0)$ to $B(\infty) = A / w_A$ crosses the center of the ellipse. Hence we get the center of the ellipse:
$C_E = \frac{1}{2}(B(t_0) + B(\infty))$
for $f_1$ we can choose the vector from the center to $B(\infty)$
$f_1 = \frac{1}{2}(B(\infty) - B(t_0)) = \frac{1}{2}(\frac{A}{w_A} - \frac{C}{w_C})$
To get $f_2$ we can use the fact that $E(\pi/2) = C_E + f_2$ and $E'(\pi/2) = f_1$. (I.e. the point at which the tangent of the ellipse is parallel to $f_1$ is equal to $C_E + f_2$). As B and E have different parameters, the derivates are different, however the derivates in the same point are parallel. I.e. we need to figure out for which $t_1$ $B'(t_1) = u f_1, u \in \mathbb{R}$ (For simplification I am assuming that $t_0 = 0$ in the following equations. If it is not, an equal Bézier with $t_0$ shifted to $0$ can be obtained by expanding $B(t + t_0)$)
$$B'(t) = \frac{t^2 (A w_B - B w_A) + 2t(A w_C-Cw_A)+(Bw_C-Cw_B)}{(t^2 w_A + t w_B + w_C)^2} = u f_1$$
As the denominator is a simple scalar, it does not change the direction of the vector and can be ignored in this case. The 1/2 scalar in $f_1$'s definition can also be ignored:
$$t^2 (A w_B - B w_A) + 2t(A w_C-Cw_A)+(Bw_C-Cw_B) = u (Aw_C - Cw_A)$$
Subtracting $2t(A w_C-Cw_A)$ from both sides does not change the direction either:
$$t^2 (A w_B - B w_A) +(Bw_C-Cw_B) = u (Aw_C - Cw_A))$$
The above equation is satisfied with $t_1 = \sqrt{w_C /w_A}$ and we can calculate:
$f_2 = B(t_1) - C_E$
I have two issues with the above solution: a) it breaks down when the weights are chosen such that $w_A = 0$, as $t_0$ is undefined. b) While the steps for the center and $f_1$ work analogously in the case of a hyperbola, I have not been able to figure out a similar condition to calculate $f_2$.
This answer for a similar question suggests converting the rational bezier into a matrix conic form, however I'm not sure this works in 3D?
I think you might have made things a bit harder for yourself by trying to work directly with the control points in 3D. Since affine transformations don’t change the type of a conic, I suggest working with a simple case in the plane and then mapping it to the control points that you have. Furthermore, I would work in homogeneous coordinates to avoid having to treat various infinities as special cases. In homogeneous coordinates, the quadratic rational Bézier parameterization becomes the simpler to work with $$R(t) = (1-t)^2w_0\mathbf p_0+2t(1-t)w_1\mathbf p_1+t^2w_2\mathbf p_2.\tag1$$ This can be viewed as the image of the parabola described by the quadratic Bézier $B$ with the same control points under the homography with matrix $$\mathtt H = \begin{bmatrix}\mathbf p_0&\mathbf p_1&\mathbf p_2\end{bmatrix} \operatorname{diag}(w_0,w_1,w_2) \, \begin{bmatrix}\mathbf p_0&\mathbf p_1&\mathbf p_2\end{bmatrix}^{-1}.$$ This homography maps the control points to themselves and the parabola’s “infinity point” $[t^2]B(t) = \mathbf p_0-2\mathbf p_1+\mathbf p_2$ to the “infinity point” $\mathbf p_\infty = [t^2]R(t) = w_0\mathbf p_0-2w_1\mathbf p_1+w_2\mathbf p_2$. I won’t be making use of this here, but it provides an alternate route to derive what follows.
We can write an implicit Cartesian equation for $R(t)$ directly by making use of Plücker’s mu: Let $\mathbf l_{ij}=\mathbf p_i\times\mathbf p_j$ be the line through control points $\mathbf p_i$ and $\mathbf p_j$. Then an equation of the conic described by $R$ is $$(\mathbf l_{02}\cdot\mathbf x)^2(\mathbf l_{01}\cdot\mathbf p_\infty)(\mathbf l_{12}\cdot\mathbf p_\infty) = (\mathbf l_{02}\cdot\mathbf p_\infty)^2(\mathbf l_{01}\cdot\mathbf x)(\mathbf l_{12}\cdot\mathbf x). \tag2$$ The equation $(\mathbf l_{01}\cdot\mathbf x)(\mathbf l_{12}\cdot\mathbf x) = 0$ represents the degenerate conic consisting of the two tangent lines while $(\mathbf l_{02}\cdot\mathbf x)^2=0$ is the double line through the outer control points. Equation (2) is a linear combination of these two conics that passes through $\mathbf p_\infty$ (which, somewhat ironically, is the only point of this conic that is not generated by $R$).
Taking the simple case of $\mathbf p_0=[1:0:1]$, $\mathbf p_1=[0:0:1]$ and $\mathbf p_2=[0:1:1]$, we have $\mathbf p_\infty = [w_0:w_2:w_0-2w_1+w_2]$ and equation (2) expands to $$w_0w_2(x+y-1)^2 = 4w_1^2xy.\tag3$$ This is an ellipse when $\Delta = w_0w_2 - w_1^2 \gt 0$, a parabola when $\Delta=0$ and a hyperbola when $\Delta\lt0$. The center of this conic can be found to be $\mathbf c = [w_0w_2:w_0w_2:2\Delta]$ using any of the usual methods. This is a point at infinity for a parabola, as expected.
For the parameterization that you want, you need endpoints of conjugate diameters. Since the tangent to the conic at $\mathbf p_0$ is the $x$-axis, if we take $\mathbf p_0$ to be the endpoint of a diameter, its conjugate diameter is the horizontal line through $\mathbf c$. The coordinates of its endpoints are found by setting $y=w_0w_2/2(w_0w_2-w_1^2)$ in equation (3) and solving for $x$ to get $$x = {w_1^2+\Delta\pm2\sqrt{w_1^2\Delta} \over 2\Delta}.\tag4$$ By symmetry, if you start with $\mathbf p_2$ instead, the above expression gives the $y$-coordinates of the endpoints of the conjugate diameter. Letting $\mathbf p$ and $\mathbf q$ be the two points you’ve chosen, we then have the parameterization $$\tilde{\mathbf c}+(\tilde{\mathbf p}-\tilde{\mathbf c})\cos t + (\tilde{\mathbf q}-\tilde{\mathbf c})\sin t\tag5$$ for the ellipse. (Here, the tilde indicates the inhomogeneous Cartesian coordinates of the point.) The above also works for a hyperbola if you take the absolute value of $\Delta$ under the radical in (4), in which case the parameterization is $\tilde{\mathbf c}+(\tilde{\mathbf p}-\tilde{\mathbf c})\cosh t + (\tilde{\mathbf q}-\tilde{\mathbf c})\sinh t$.
Finally, you need to affinely map this to the original control points $P_0$, $P_1$ and $P_2$, which can be achieved via $(x,y)\mapsto P_1+(P_0-P_1)x+(P_2-P_1)y$. For the center of the conic this gives $\tilde{\mathbf c}'=P_1+{w_0w_2\over2\Delta}(P_0-2P_1+P_2)$ and I’ll leave working out the images of the two vectors to you.