I have two triangles in 3d. I need to calculate transformation matrix(3X3) between two triangles in 3D.
1)How can I calculate the transformation matrix(rigid) while fixing one of the points to the origin(to get rid of the translation part)?
2)How does it affect if the deformation is non rigid?
On
Let the two triangles be $(O,A,B)$ and $(O,A', B')$, put ${\bf a}:=OA$, $\,{\bf b}:=OB$, and similarly for $A'$, $B'$. Then you are looking for a linear transformation $T:\ {\mathbb R}^3\to{\mathbb R}^3$ such that $T{\bf a}={\bf a}'$, $\ T{\bf b}={\bf b}'$. In the first place such a transformation is not uniquely determined. In ${\mathbb R}^3$ you can prescribe the images ${\bf a}_i'$ of three linearly independent vectors ${\bf a_i}$ in order to determine a linear map $T$ uniquely.
Maybe there are extra circumstances present in your situation. For instance, the two triangles might be euclidean congruent to begin with, and $T$ should be a rotation. In this case the conditions $$T{\bf a}={\bf a}', \quad T{\bf b}={\bf b}', \quad T({\bf a}\times{\bf b})={\bf a}'\times {\bf b}'$$ determine a certain rotation $T$ (whence the matrix of $T$ with respect to the standard basis) uniquely. To prove this, consider the three vectors ${\bf a}$, ${\bf b}$, ${\bf a}\times{\bf b}$ as a new basis of ${\mathbb R}^3$.
1) The transformation matrix is in general 4x4: \begin{equation} \begin{bmatrix} R&t\\ 0&1\\ \end{bmatrix} \end{equation} with $R$ being a 3x3 rotation matrix and $t$ being a translation vector.
Let $x_1,x_2,x_3\in\mathbb{R}^3$ be the first triangle and $\bar{x}_1,\bar{x}_2,\bar{x}_3\in\mathbb{R}^3$ the second one.
Then following Arun et al. 1987, we get:
\begin{equation} c = \frac{1}{3}(x_1 + x_2 + x_3), \quad \bar{c} = \frac{1}{3}(\bar{x}_1 + \bar{x}_2 + \bar{x}_3), \quad\text{(calculate centroids)} \end{equation}
\begin{equation} y_i = x_i-c, \quad \bar{y}_i = \bar{x}_i-\bar{c}, \quad\text{(subtract centroids from points)} \end{equation}
\begin{equation} H = y_1\bar{y}_1^\top + y_2\bar{y}_2^\top + y_3\bar{y}_3^\top, \quad U\cdot\Sigma\cdot V^\top = H \quad\text{(singular value decomposition)} \end{equation}
\begin{equation} R = V U^\top, \quad t = \bar c - Rc \end{equation}
2) The solution above is a least-squares fit, so if there is a tiny deformation this should not affect the result too much. It is hard to say more about it, if we don't know what kind of deformation we are expecting (scale change only, affine deformation, or something else).