Shouldnt $\begin{bmatrix} x \\ y \\ z \\ \end{bmatrix} $ from both equations be the same?
eq(1)$\begin{bmatrix} x \\ y \\ z \\ \end{bmatrix} = \begin{bmatrix}r_{11} & r_{12} & r_{13} \\r_{21} & r_{22} & r_{23} \\r_{31} & r_{32} & r_{33} \end{bmatrix}\begin{bmatrix} X \\ Y \\ Z \\ \end{bmatrix} + tvec $
eq(2)$ \begin{bmatrix} x \\ y \\ z \\ \end{bmatrix} = \begin{bmatrix}r_{11} & r_{12} & r_{13} & t_x\\r_{21} & r_{22} & r_{23} & t_y\\r_{31} & r_{32} & r_{33} & t_x\end{bmatrix}\begin{bmatrix} X \\ Y \\ Z \\ 1 \\ \end{bmatrix}$