Get 3d ray in world coordinates from a 2d pixel location

58 Views Asked by At

I am trying to project a 3d ray from a 2d image pixel location. I have the actual 3d coordinates of a pixel location and want the ray from the camera to intersect this point but currently, my line is missing this point, see Figure 1

I am using the perspective projection equation See here. I am able to successfully convert 3D points to pixel locations but it fails when I try to invert the equation to project the 3d ray. I was expecting the ray line in the direction from the camera centre to the inverse 3d point to intersect the original 3d point but it misses it.

I have seen similar question here but when I try their code it gives the same problem. I am not sure where I am going wrong, any help would be appreciated

Thank You!

My Python code is below.

import numpy as np
import matplotlib.pyplot as plt
from numpy.linalg import inv, pinv

#########################################################
#Important Constants
p_w = np.array([[-410, 0, 10, 1]]).T                            # world point (4x1), (x,y,z,1)
K = 1.0e+03*np.array(                                           # Camera intrinsics (3x3) 
                         [[1.5665, 0,1.0323],
                         [0, 1.5693, 0.7825],
                         [0, 0, 0.0010]], dtype = "double"
                         )
R = np.array(                                                   # R_world (3x3)
                [[-0.92232435, 0.00751749, 0.38634348],
                [-0.14904685, 0.9155222, 0.37363637],
                [0.35651484, 0.4021972, -0.84328796]], dtype = "double"
                )


t = np.array([ [-304.79755558],  [-980.59230844], [-1920.36050988]]) # T_world 
q = np.hstack((R,t))                                                 # (R|t) Transform (3x4)
C = -np.matrix(R).T @ np.matrix(t)                                   # Centre of Camera

#Convert 3D point to pixel coordinates 
p_c = np.matmul(q, p_w)                                              # world point in camera frame
p_homo = np.matmul(K, p_c)                                           # world point in image frame in homo coordinates 
p_img = p_homo/p_homo[2]                                             # world point in image frame in pixels

#Convert pixel coordinates to 3D ray 
p_c = np.matmul(inv(K), p_img)                                       #image point in camera frame
p_c = np.append(p_c, 1)                                              
pts = np.matmul(p_c, pinv(q))                                        #image point in world frame using psuedo inverse in homo coord
pt_w = pts / pts[2]                                                  #image point in world frame in mm


#plot results

fig = plt.figure()
ax = fig.add_subplot(projection='3d')
ax.scatter(p_w[0], p_w[1], p_w[2], color="g", label ="actual_3d_pt") # actual 3d point
ax.scatter(C[0], C[1], C[2], color="y", label="Camera") # camera centre

x = np.append([C[0]], [pt_w[0]])
y = np.append([C[1]], [pt_w[1]])
z = np.append([C[2]], [pt_w[2]])
ax.plot(x,y,z, label="3d-ray", color="r")               #plot line from camera origin to point on ray

ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Z Label')
ax.legend()

plt.show()