Is it possible to point one camera to the location of some coordinates on the picture taken by another camera?

Question

Let's say we have a scenario like this:

There is the pink camera and the blue camera. Both cameras have different fields of view and orientations. The pink camera takes a picture like this one:

On this 2D image that the pink camera captured, I know exactly the coordinates of the center of the green circle. Is it possible to then calculate an approximation of the orientation that the blue camera should be in to be pointing directly at the green circle? How accurate can that be?

To summarize, this is the data we have:

• The position of each camera
• Both camera's field of view size and proportion
• The orientation that the pink camera was in when it captured the image
• The location of the center of the green circle inside the image

Edit: I should make clear that what I'm asking for is an approximation, specifically on the assumption that we would be considering the surface as just a plane, all the objects have a very insignificant heights. The heights on the illustrations are just for a better visualization, including the green sphere. I apologize for not making this more clear before.

Answer 1

As John Hughes explained in his answer, what you want to do is not possible given only the information that you first provided in your question. However, with the additional information about the plane that the sphere is sitting on and the assumption that it is small relative to its distance from the cameras, the problem becomes soluble. Basically, you just need to compute the intersection of the back-projection of an image point with that plane.

Here’s one way to do this: From the information that you have about the pink camera, you’ll need to work out its $3\times4$ projection matrix $\mathtt P = [\mathtt M\mid\mathbf p_4]$. Working in homogeneous coordinates, you know two points on the back-projection of an image point $\mathbf x$: the camera’s center $\mathbf C$ and $\mathtt P^+\mathbf x$, where $\mathtt P^+$ is the right pseudo-inverse of $\mathtt P$: $$\mathtt P^+=\mathtt P^T(\mathtt P\mathtt P^T)^{-1}.$$ If the “ground” plane has equation $ax+by+cz+d=0$, then setting $\Pi=(a,b,c,d)^T$, the intersection of the back-projected ray and this plane is $$((\mathtt P^+\mathbf x)^T\Pi)\mathbf C-(\mathbf C^T\Pi)(\mathtt P^+\mathbf x). \tag1$$ If you’re not comfortable working with homogeneous coordinates, you can instead compute the intersection of the ground plane with the line through the pink camera center that has direction vector $\mathtt M^{-1}\mathbf x$.

Once you have the coordinates of this intersection point, working out the direction in which to point the blue camera is a simple exercise. It won’t point directly at the center of the green sphere, but if your assumption that it is small relative to its distance from the cameras, the error will also be small.

Answer 2

You can calculate the center of the blue camera's view and make it equal to the center of green sphere. If it's not achievable in your program you also can evaluate the center of the blue camera's view and change the heights or tilt angle of the blue camera until the center values match.
EDIT
Guess I understood the problem further. If we create 3d coordinate system the coordinates of one dimension must be the same for centers to match, so that we can create a 2d diagram like so: Thus the Z coordinates of sphere and camera are equal. From here you can evaluate the angle A (or whatever you need) to point camera on the right place. Hope this helps.

Answer 3

Consider the following:

Camera 1: looks towards origin from $(20,0,0)$

Camera 2: Looks toward origin from $(0, 20, 0)$; the image of the $x$-axis of 3-space becomes, in image $2$, the $x$-axis of image 2.

They have the same field of view, etc.

We place (in 3D) a green sphere which then appears as a green dot at the center of the camera 1 image. The location of this green sphere in 3D is at $(x, 0, 0)$ for some $x$. That's all you can know from the information you have about camera $1$.

The position of the dot in the camera 2 image is ... somewhere along the $x$-axis of that image. It could be anywhere.

So from the information given, the answer is "no, you cannot determine the orientation of camera 2 that will place the green dot at the image center," because you do not know the "depth" of the green dot (relative to camera 1), which is essential for determining the green sphere's image location in camera 2.