I am taking a university level course in discrete geometrics and graphical programming, and I'm having trouble understanding this exercise.
Let $\boldsymbol{p}$ be a point in $\mathbb{R}^3$, $\boldsymbol{n}$ a surface normal, and $\boldsymbol{M}$ a $4×4$ non-singular homogenous transformation matrix.
Using a right hand coordinate system:
How do we apply $\boldsymbol{M}$ on $\boldsymbol{n}$ to find the transformed surface normal $\boldsymbol{n}'$? Why?
Any help would be very appreciated. Thanks in advance.
Let's stick to homogeneous coordinates for now, i.e. $p\in\mathbb{RP}^3$ is a vector of four real numbers, and likewise for $n$. $M$ is a matrix which maps points, i.e. $p\mapsto p'=M\cdot p$. A point $p$ lies on a plane $n$ if its dot product with the vector of the plane is zero, i.e. if $\langle p,n\rangle=0$. The transformation you want to describe should preserve incidence, so if $p$ lies on $n$ then $p'$ should lie on $n'$:
$$\langle p',n'\rangle = p'^T\cdot n' = (M\cdot p)^T\cdot n' = p^T\cdot M^T\cdot n' \overset!=p^T\cdot n=\langle p,n\rangle$$
Strictly speaking this is not exactly the requirement: the two sides don't have to be equal, the only have to be zero for the same points. But making them equal is the easiest approach, at least conceptually. In order to satisfy this equation exactly, one would choose
$$n'=\left(M^T\right)^{-1}\cdot n$$
since in this case the two matrices cancel and the two expressions are equal. If you want to avoid divisions, you might also compute the adjunct matrix instead of the inverse, since these only differ by a scalar factor. Since your problem statement guarantees that $M$ is non-singular, the inversion is always possible.
Now back to your assumed non-homogeneous input. The vector $n$ you actually have is just a normal vector in $\mathbb R^3$. To turn that into the vector of a plane in $\mathbb{RP}^3$ one would add a fourth coordinate, which describes the offset from the origin. Then you'd transform that, and the first three coordinates of that result are the normal vector of the resulting plane.
If your transformation $M$ is affine, then the entries $m_{41}$ through $m_{43}$ will be zero. As a consequence of this, the last column of $(M^T)^{-1}$ will be all zero except for the last entry. In this case you don't actually need the offset: simply take the upper left $3\times 3$ submatrix of the transpose inverse (or transpose adjunct) and apply that to your normal vector $n$.