I have a plane of 3d points. The plane has normal pointing in $(n_x, n_y, n_z)$.
The plan is to align the plane to the direction of $(1,0,0)$ via 2 transformations:
1) Rotate about z-axis for angle $-\gamma$, where $\gamma = \tan^{-1}(\frac{n_y}{n_x})$, giving me a rotation matrix $R_1$, and
2) Rotate about y-axis for angle $\beta$ where $\beta = \tan^{-1}(\frac{n_z}{\sqrt{(n_x ^ 2 + n_y^2}})$, giving me a rotation matrix $R_2$
As per https://en.wikipedia.org/wiki/Rotation_matrix:
$R_1 = {\begin{bmatrix}\cos \gamma &\sin \gamma & 0 \\ -\sin \gamma&\cos \gamma & 0 \\ 0&0 & 1 \\\end{bmatrix}} $
$R_2 = {\begin{bmatrix}\cos \beta&0& \sin\beta \\ 0&1 & 0 \\ -\sin\beta&0 & \cos\beta \\\end{bmatrix}} $
My question is that, if I want to transform a point $P$ that is lying on the original plane, do I have to normalize $P$ in order to use this equation:
$P' = R_2 * R_1 * P$
where $P'$ is the point lying on the transformed plane. If so, why?
As Chris Cutler wrote in his comment, why would you want to normalize $P$? The rotations just work.
That aside, you don’t need to compute any trig functions or their inverses in order to build your rotation. Using the identities $\tan\theta={\sin\theta\over\cos\theta}$ and $\cos^2\theta+\sin^2\theta=1$, you can compute the sines and cosines of the rotation angles directly from the plane’s normal $\mathbf n$: $$\cos\gamma={n_x\over\sqrt{n_x^2+n_y^2}}, \sin\gamma={n_y\over\sqrt{n_x^2+n_y^2}}$$ and $$\cos\beta=\sqrt{n_x^2+n_y^2}, \sin\beta=n_z,$$ assuming, of course, that $n_x^2+n_y^2\ne0$. If it is, then you don’t need the first rotation, anyway, because $\mathbf n$ is aligned with the $z$-axis. You can also combine the two rotations into a single matrix, which might be a good thing to do for computational efficiency if you’re going to be transforming several points.
However, there’s a more direct way to construct the combined rotation matrix using the facts that the columns of a transformation matrix are the images of the basis vectors and that the inverse of a rotation matrix is its transpose. You want $\mathbf n$ to end up aligned with the $x$-axis, so the first row of the matrix is just $\mathbf n$. It’s not too hard to work out that the first rotation sends the vector $(-n_y,n_x,0)$ to the $y$-axis, so the second row of the matrix will be this vector, normalized. The last row is the cross product of the first two. Putting this all together, you get $$R=\begin{bmatrix}n_x&n_y&n_z \\ -{n_y\over\sqrt{n_x^2+n_y^2}}&{n_x\over\sqrt{n_x^2+n_y^2}}&0 \\ -{n_xn_z\over\sqrt{n_x^2+n_y^2}}&-{n_yn_z\over\sqrt{n_x^2+n_y^2}}&\sqrt{n_x^2+n_y^2}\end{bmatrix}.$$ I’ll leave it to you to verify that this is indeed what you get when you multiply the two individual rotation matrices together. Note, too, that you can simplify the computations a little bit by substituting $1-n_z^2$ for $n_x^2+n_y^2$.