Recovering the covariance matrix of pixel coordinate from normalised camera coordinate

293 Views Asked by At

lets assume we have the pixel value of some interest points as well as their covariance matrices that defined as follows: $$ \mathbf x^{\prime} = \begin{bmatrix} x^{\prime}\\ y^{\prime}\\ \end{bmatrix} , \mathbf \Sigma_{x\prime x\prime} = \begin{bmatrix} \sigma^2_{x^{\prime}} & \sigma_{x^{\prime} y^{\prime}}\\ \sigma_{x^{\prime} y^{\prime}} & \sigma^2_{y^{\prime}} \\ \end{bmatrix} $$ after that, pixel points convert to homogenous coordinate from Euclidean coordinate and are projected to the normalised camera coordinate by $K^{-1}$, where K represents the camera matrix.

$$ \mathbf x = K^{-1} \mathbf x^{\prime} = \begin{bmatrix} x\\ y\\ 1\\ \end{bmatrix} $$ in the next step, we compute the covariance matrices for each new $\mathbf x$ point. $$ \mathbf \Sigma_{\mathbf x \mathbf x} = J_{\pi} \mathbf \Sigma_{x^{\prime} x^{\prime}} J_{\pi}^{T} = \begin{bmatrix} \sigma^2_{x} & \sigma_{x y} & 0\\ \sigma_{x y} & \sigma^2_{y} & 0\\ 0 & 0 & 0\\ \end{bmatrix} $$ where $J_{\pi}$ is the jacobian of forward projection. We also know the rank of matrix $\mathbf \Sigma_{\mathbf x \mathbf x}$ is 2 and it's singular and invertable.
Next, we compute the normalised point $\mathbf x$ and we called it $\mathbf v$. $$ \mathbf v = \begin{bmatrix} v_x\\ v_y\\ v_z\\ \end{bmatrix} = \frac {\mathbf x}{\Vert \mathbf x \Vert}, \mathbf \Sigma_{\mathbf v \mathbf v} = \begin{bmatrix} \sigma^2_{v_x} & \sigma_{v_{xv}} & \sigma_{v_{xz}} \\ \sigma_{v_{xz}} & \sigma^2_{v_y} & \sigma_{v_{yz}} \\ \sigma_{v_{zx}} & \sigma_{v_{zy}} & \sigma^2_{v_z}\\ \end{bmatrix} $$ where $$ \mathbf \Sigma_{\mathbf v \mathbf v} = J \mathbf \Sigma_{\mathbf x \mathbf x} J^{T}, J = \frac{1}{\Vert \mathbf x \Vert}(\mathbf I_{3} - \mathbf v \mathbf v^{T}) $$ We also know the $\mathbf \Sigma_{\mathbf v \mathbf v}$ remains invertable. From now, I can compute the rest procedure and update $\mathbf \Sigma_{\mathbf v \mathbf v}$.
My main question is, after updating the $\mathbf \Sigma_{\mathbf v \mathbf v}$, how can I update the $\mathbf \Sigma_{\mathbf x \mathbf x}$ and $\mathbf \Sigma_{\mathbf x^{\prime} \mathbf x^{\prime}}$ sequentially. In fact, the main difficult part of this job is extracting the $\mathbf \Sigma_{\mathbf x \mathbf x}$ from updated $\mathbf \Sigma_{\mathbf v \mathbf v}$ while we should keep the rank of $\mathbf \Sigma_{\mathbf x \mathbf x}$, 2 and set the last column and row of that $0$ (similar to formula).
It is noticeable, all equations, which I mentioned here, come from MLPnP reference and here, I just summarised the equations 2-6.