I want to retrieve the sign of scaling for each axis from modelview matrix. Right now I am able to extract the sign only if all 3 signs are same but it fails when one of them is different. Here is the example I am trying:
float []mat={0.032254f, 0.000000f, 0.000000f, 0.000000f, 0.000000f, -0.0052254f, 0.000000f, 0.000000f, 0.000000f, 0.000000f, 0.4332254f, 0.000000f, 0.000000f, 0.000000f, 0.000000f, 1.000000f};
The code is:
glm::vec3 Xaxis(matrix[0][0],matrix[0][1],matrix[0][2]);
glm::vec3 Yaxis(matrix[1][0],matrix[1][1],matrix[1][2]);
glm::vec3 Zaxis(matrix[2][0],matrix[2][1],matrix[2][2]);
double zs=glm::dot(glm::cross(Xaxis,Yaxis),Zaxis);
double ys=glm::dot(glm::cross(Zaxis,Xaxis),Yaxis);
double xs=glm::dot(glm::cross(Yaxis,Zaxis),Xaxis);
xs=std::signbit(xs);
ys=std::signbit(ys);
zs=std::signbit(zs);
You can use $$ s_i = \text{sgn}(m_{ii}) \quad (i \in \{1,2,3\}) \quad (*) $$ if $M=(m_{ij})$ is a scaling matrix.
If your matrix is $$ M = \left( \begin{array}{rrrr} R_{11} S_1 & R_{12} S_2 & R_{13} S_3 & T_1 \\ R_{21} S_1 & R_{22} S_2 & R_{23} S_3 & T_2 \\ R_{31} S_1 & R_{32} S_2 & R_{33} S_3 & T_3 \\ 0 & 0 & 0 & 1 \end{array} \right) $$ then try $$ S_i = \sqrt{(M^T M)_{ii}} $$ first. And then use $S_i$ instead of $m_ii$ in $(*)$.