How to determine the pose of a 3D ellipse given two projections to 2D of the ellipse?

Question

This question is about whether I'm doing an optimization problem the right way, or if it can be done in a simpler way. I'll start with my problem scenario and then move into my proposed solution. I would like to know if my proposed solution is the best way to proceed.

Problem

I have an ellipse in 3D that has been projected to two cameras.

Let's say this is the image of the ellipse from camera 1: ellipse from camera 1

It has these parameters in the image: {"cx":264,"cy":158,"rx":18,"ry":4.243,"theta":-0.785}

And this is the image of the ellipse from camera 2: ellipse from camera 2

It has these parameters in the image: {"cx":376,"cy":131,"rx":20,"ry":17,"theta":3.024}

And the "real-life ellipse" (front-on view of the ellipse) looks like this (note in my scenario it's meant to be a 2D ellipse but in 3D space, as opposed to an ellipsoid): ellipse in 3D

It has these parameters in real-life (somewhat arbitrary, aka not an actual calculation): {"cx":3,"cy":5, "cz":1,"rx":18,"ry":30,"rz":0}

I'd really like to find a way to get the parameters of the real-life ellipse using just the parameters of the ellipses from the pair of images. In effect, to triangulate the ellipse in 3D from the pair of projections. Assume no point correspondences are known (that's what makes this specific problem tricky).

Known Information

• I have the extrinsic and intrinsic camera parameters for both cameras. I therefore have the projection matrices that project any object in 3D to the image plane for each camera.

Thoughts

I'm thinking about using something like the Iterative Closest Point algorithm, where I

1. Take an initial guess for the parameters of the ellipse in 3D
2. Project the 3D ellipse to each camera. By this I mean that I would project about 10 equidistant points from the ellipse in 3D to 2D.
3. Get the total error.

• For each image, determine the error between the projected ellipse and the ellipse in the image. By this, I am thinking of fitting an ellipse to the 10 projected points, retrieving the ellipse parameters, and then computing the difference between the sum of the squares of the parameters: $$error = \sum_{i}^{k} i^2 - \sum_{j}^{k} j^2$$

  where $i$ represents the parameters in the ellipse of the image and $j$ represents the parameters in the projected ellipse.

• Add the errors from the two images to get the total error.

4. If the error is low enough, stop.
5. Compute a transformation for the ellipse in 3D, then go back to step 2.

Question

How in the world can I compute the transformation from step 5?

I'm thinking that I could just take the derivative of the error function. But the error function seems like it would be really complex, because it would entail all the operations of:

• projecting 3D to 2D
• fitting an ellipse to the projected points
• taking the sum of squares of the ellipse parameters
• doing that for both images and adding the results together

Is there a simpler way to do what I'm trying to do?

I appreciate any help. I hope this is a clear post, but please let me know if there's any information I can clarify or add.

Thanks!

Answer 1

Let’s call the projected ellipses $P1$ and $P2$, and the 3D ellipse $E$.

The ellipse $P1$, swept along its camera viewing vector gives you an elliptical cylinder, $C1$.

Take 5 points on ellipse $P2$. Sweeping these points along the corresponding camera viewing vector gives you five 3D lines.

Intersect each of the five lines with cylinder $C1$ to get 5 points. The 3D ellipse $E$ that you want must be the one that passes through these 5 points (roughly speaking, see below for complications).

So, the basic required computation is intersecting a straight line with an elliptical cylinder. This just requires solution of a quadratic equation, so easy enough. But, of course, each equation has two solutions (because the line intersects the cylinder in two places). So, actually, you’ll get 10 intersection points, not 5. You will find that 5 points lie on one plane, and the other 5 lie on a different plane, so dividing the 10 points into two groups of 5 should be easy. Then, each set of 5 points gives you an ellipse, so in fact there are two possible 3D ellipses $E$ that have $P1$ and $P2$ as their projections.

All of the above assumes that you’re using parallel projections (cameras at infinity). If you’re using perspective projections, then use an elliptical cone instead of a cylinder.