Multiple objects triangulation in 3D, intersecting the right vectors (rays)

Question

I am working on a project in which I should be able to triangulate the position of multiple objects when they are seen by (at least) two cameras.

Single object

Currently I am able to triangulate a single object. This is done in some steps:

1. For every camera, it is calculated the vector pointing to the pixel where object's center lies. This is done by properly rotating the vector pointing at the center of the image (0,0,1) on Y and X axis.
2. Then every direction vector is converted from camera's own coordinates to world coordinates (through glm math library) in order to have a common coordinate system to perform calculus.
3. Endpoints of the minimum length segment between each couple of rays are computed. A ray here is defined as position of the camera + λ * direction vector with λ >= 0.
4. The average of segments' central points is the approximate object's position in the world.

Multiple objects

My problem with multiple objects is that I can't find a proper way to determine if a pair of rays are pointing the same object, so they should be matched, or not. I draw an example of this.

I think I am close to a solutions but I am still missing the last piece. My partial solution is empirical and works with the various angles involved, in particular I use the following four angles:

• α: angle between directions of the rays
• β1, β2: angle between camera's direction and own ray's direction (x2)
• δ: angle between the directions of the cameras

Here's a drawn example. The relation I found is:

α + β1 + β2 - δ = 0

I made some tests and sometimes it works, sometimes it would if changing the sign of β1 and/or β2. The problem is that I don't know how to determine when. I have an additional information to use, the (x,y) angles used in step 2 to produce the direction of the ray from (0,0,1).

These two values are signed and strictly related to βn, because (0,0,1) is basically the direction of the camera in camera's own coordinate system and the same is true for the computed ray direction. So I think they will determine the sign of β, but I don't know how.

Any idea on how to finish it? Is this model mathematically correct?