I am not entirely sure that this is the right place to ask or if my problem is solvable, but I am out of ideas and googling hasn't brought me any closer to a solution unfortunately. Again if anyone knows of a better place to ask the question, feel free to point me there, Thanks.
I am trying to find a way to create a link between a 2D image and 3D depth data and consider that the 2D perspective warps the objects in the image as they move away from the center.
Example: An average 0.5L soda bottle shot from a top view with a camera will look different when it is directly underneath the camera and when it is at the border of the cameras FOV. I assume that this is called perspective warp? The camera can also capture depth information. When the bottle is underneath the camera the depth data will show the cap of the bottle along with the curve leading down to the side of the bottle. If the bottle is at the edge of the cameras FOV the depth data will show the cap and the curved side of the bottle that is closest to the camera.
When the bottle is underneath the camera, the coordinates of the bottle edges will be approximately the same in the image and the depth data, but when the bottle is at the edge of the cameras FOV the bottle edges will be "larger" than what it is in the depth data.
How can I consider that the object in the image gets skewed as I move away from the camera center? I know the FOV of the camera and the height of the object if that is useful.
Any help is much appreciated.
Edit: So the past few days I have been looking into the camera matrices and gathered the following I have a camera with a focal length of 1.93mm according to the datasheet. The sensor FOV is HxV 90x65 The image captured is 1280 x 720px, so the center is 640x360px The sensor pixel size is 3x3um I am unsure of how to calculate fx, fy since I have found two formulas:
fx = F / px = 1.93 / 1280 = 0.0015078125
fx = (px / 2) / tan(FOVx / 2) = 640 / tan(45) = 640
cx = 640
cy = 360
z = 650 fixed distance
From these informations, I can create my camera matrix with either
I am still very unsure about how to go about transforming my 2D coordinate into 3D with the matrix and also not sure if I am supposed to use the extrinsic matices as well since the camera orientation is straight down the rotation matrix is just zeros and the camera is centered in the world space, so the transformation matrix would also me zeros i assume?
Best regards Martin Pedersen