Rotating a plane defined by a normal and a distance from the origin around an arbitrary point in 3D space

Question

I have a plane defined by its normal and its distance from the origin. I have a rotation matrix and a point in 3D space around which to do the rotation.

What formula will allow me to do the rotation?

Answer 1

Let the equation of the plane be $$ \textbf{a}^T\textbf{x} = b $$ where $\textbf{a}$ is a unit vector normal to the plane and b is the plane's distance from the origin.

Transform to a coordinate system centred on a point $\textbf{r}$: $$ \textbf{x}' = \textbf{x} -\textbf{r} $$ Then in the dashed coordinate system: $$ \textbf{a}^T(\textbf{x}' + \textbf{r}) = b \\ \textbf{a}^T\textbf{x}' = b - \textbf{a}^T\textbf{r} $$ In this coordinate system apply the rotation to $\textbf{a}$: $$ \textbf{a}^T\,\textbf{M}^T\textbf{x}' = b - \textbf{a}^T\textbf{r} $$ Then transform back to get the equation of the rotated plane in the original coordinate system: $$ \textbf{a}^T\textbf{M}^T(\textbf{x} - \textbf{r})= b - \textbf{a}^T\textbf{r} \\ \textbf{a}^T\textbf{M}^T\,\textbf{x} = b - \textbf{a}^T\textbf{r} + \textbf{a}^T\textbf{M}^T\,\textbf{r} $$