I have a 3D scene with a perspective camera in it. I want to make a 2D object in the scene to rotate on X axis and Y axis to always look in the direction of the camera so from the Camera point of view it won't be visible as a 2D object.
I can get the values of camera X, Y and Z position and use them to calculate the X and Y rotation but I am don't know what maths I need to do to calculate them.
Attach a frame of reference to the object such that the object completely lies in the $x'y'$ plane of this frame.
And attach a frame of reference to the perspective camera, with the origin of the frame at the focal point of the camera, with its $x'', y''$ axes pointing in the horizontal and vertical directions of the image the camera generates, and its $z''$ is determined by the cross product of the unit vectors along $x''$ and $y''$.
If $p$ is any point in space, then
$ p = O_c + R_c p_c = O_o + R_o p_o $
where $p$ is the coordinate vector with respect to the world frame, $p_c$ is the coordinate vector of the same point, but with respect to the camera frame, and finally, $p_o$ is the coordinate vector of the same point, but with respect to the object frame. $O_c$ is the position of the focal point of the camera with respect to the world frame, and $O_o$ is the position of the origin of the object's reference frame with respect to the world frame.
If we now apply a rotation to the object about a chosen point $q$ that is specified in world coordinates, by a rotation matrix $R$, then the image of a point $p$ will be $p'$ given by
$ p' = q + R (O_o + R_o p_o - q) = (q + R(O_o - q) ) + R R_o p_o $
The line connecting $p'$ to $O_c$ is
$ r(t) = O_c + t (p' - O_c) $
We want this line to be contained in the plane (given in world coordinates):
$n^T ( r - (q + R (O_o - q) ) ) = 0 $
where $n$ is the third column (the z' axis) of the matrix $R R_o$.
Substitute $n$ and $r(t)$ , you get
$ ( R R_o e_3 )^T ( O_c + t ( q - O_c + R (O_c - q) ) - (q + R (O_o - q) ) ) = 0 $
where $e_3 = [0, 0, 1]^T $
This simplifies to
$ e_3^T R_o^T R^T ( (1 - t) O_c + (t - 1) (q + R (O_o - q) ) ) = 0 $
Since this equation is to hold true for any real $t$, then, this implies that we must have
$ e_3^T R_o^T R^T ( (q + R (O_o - q)) - O_c ) = 0 $
And this re-arranges to
$ e_3^T R_o^T ( O_o - q + R^T (q - O_c) ) = 0 $
Now, define $v = O_o - q $ and $ w = q - O_c $
Further, let $ v = R_o v' $ and $ w = R_o w' $, then our equation becomes,
$ e_3^T R_o^T ( R_o v' + R^T R_o w' ) = 0 $
And this simplifies to
$ e_3^T ( v' + R_o^T R^T R_o w' ) = 0 $
Let $R' = R_o^T R^T R_o $, then
$ e_3^T (v' + R' w') = 0$
i.e.
$ e_3^T (R' w') = - e_3^T v' $
This equation means that we have to choose the rotation matrix $R'$ such that $w'$ is rotated to a vector $w'_1 = R' w'$ such that the $z$ entry of $w'_1$ is equal to $(- v'_z) $.
Hence,
$ w'_1 = ( a \cos \phi, a \sin \phi , - v'_z ) $
where $\phi \in [0, 2\pi)$ is arbitrary.
where
$ a = \sqrt{ \| w' \|^2 - (v'_z)^2 } $
So set $\phi$ to some value of your choice, and calculate $w'_1$. Now you have to calculate the rotation matrix that will send $w'$ to $w'_1$. There is an infinite number of these rotation matrices, the simplest on, is where the axis of rotation is
$ a = \dfrac{ w' \times w'_1 }{\| w' \times w'_1 \| } $
And the angle of rotation is the angle between $w'$ and $w'_1$.
Now the rotation matrix $R'$ can be computed using its axis and angle of rotation using the Rodrigues' rotation matrix formula.
Having calculated $R'$, we can now calculate $R$ from
$ R' = R_o^T R^T R_o $
So that
$ R^T = R_o R' R_o^T $
and therefore,
$ R = R_o R'^T R_o^T $
We're done!!
That's all to it.
I've tested this calculation with an object that is a square with center at $(0, 15, 50)$ , and a reference frame that is of the form $R_o = R_z(25^\circ) R_y(30^\circ) R_z(25^\circ) $.
A camera reference frame is generated from the focal center at $(-50, -50, -100) $ and a point along its axis of view equal to $(20, 30, 250)$.
The anchor of rotation is chosen as the point
$q = \dfrac{3}{4} O_o + \dfrac{1}{4} O_c $
The view of the object before rotation is provided in the first picture, followed by the view after applying the rotation.