I have a particle in a 3D space, I'm projecting it into the image space by using $$x=PX$$ Where $x$ is the image space, $P$ is the projection Matrix, and $X$ is the world space location. I'm trying to apply a gradient descent optimization by calculating the image space difference. I'm doing that in 2D without any problems. But with 3D, everything is going wrong because of the Z coordinate, Can you please help me to understand backpropagating the image space to the 3D world space for X, Y, and Z? Thanks.
Update: What I'm doing currently is calculating the Jacobian Matrix $M$ of the projection matrix $P=[M\mid\mathbf p_4]$ and the Jacobian matrix of the image space, and multiplying them as follow: $$\begin{bmatrix}\frac1w&0&-\frac x{w^2} \\ 0&\frac1w&-\frac y{w^2}\end{bmatrix} \begin{bmatrix} a_{11}&a_{12}&a_{13} \\ a_{21}&a_{22}&a_{23} \\ a_{31}&a_{32}&a_{33} \end{bmatrix} = \begin{bmatrix} {a_{11}w-a_{31}x \over w^2} & {a_{12}w-a_{32}x \over w^2} & {a_{13}w-a_{33}x \over w^2} \\ {a_{21}w-a_{31}y \over w^2} & {a_{22}w-a_{32}y \over w^2} & {a_{23}w-a_{33}y \over w^2}\end{bmatrix}.$$ I have a problem taking these values and applying them to my particle 3D position.