I am using quaternions to describe 3D transformations. A position in space is representated by a (x,y,z,1) vector, and a transformation by a 4x4 matrix, following quaternions logics as far as I could understand it.
Is there a name for 4 x 4 matrix obtained by combinations of translations and rotations, more precisely a spatial transformation that conserves lengths? (no scaling, no projection) Is there a formula to identify such transformations on any 4x4 matrix?
It's called a "rigid motion". To detect whether a 4x4 matrix, operating on homogeneous $4 \times 1 $ coordinate tuples, represents a rigid motion...
Check that the bottom row is $\begin{bmatrix}0& 0& 0 &s\end{bmatrix}$, with $s \ne 0$.
Let $A$ denote the upper left $3 \times 3$ matrix, and compute $A^T A$; if it's not $s^2 I$, then you don't have a rigid motion. If it is, then you have a rigid motion.
Check $\det A > 0$; if that's true, then you have an orientation-preserving rigid motion; otherwise, you've got an orientation-reversing one (but following it by reflection through the $xy$-plane, say, will make it orientation-preserving.