From a set of N points $\{X_i,Y_i,Z_i\}_{i=0}^N$ I have fitted a plane: $$ Z = a + bX + cY $$
However the points were measured by an instrument that unfortunately was slightly rotated with a $(roll,pitch,yaw)$ wrt to the "world" coordinate system.
$$ \begin{bmatrix} X \\ Y \\ Z \end{bmatrix} = R \begin{bmatrix} X_W \\ Y_W \\ Z_W \end{bmatrix} $$ where $R(roll,pitch,yaw)$ is an rotation matrix.
Given a set of plane coefficients: $(a, b, c)$ and a rotation matrix R, how do I find the plane coefficients in the unrotated world coordinates?
So far I have "unrotated" all points: $$ \begin{bmatrix} X_W \\ Y_W \\ Z_W \end{bmatrix} = R^{-1} \begin{bmatrix} X \\ Y \\ Z \end{bmatrix} $$ and fitted a new plane.
My problem is that I am doing parameter estimation of (roll, pitch, yaw). So for each Square Error minimization loop I have to "unrotate" all points and fit a new plane.
I would prefer to only fit a plane once and then somehow "rotate" the plane coefficients.
You can also write your plane as $bX+cY-Z+a=0$. Using homogeneous coordinates for the points you get a point lying on a plane as
$$[b,c,-1,a]\cdot\begin{bmatrix}X\\Y\\Z\\1\end{bmatrix}=0$$
This product of a row vector and a column vector is essentially just a dot product, but written in matrix notation. Now we can include your rotation $R$, padded with one row and one column to deal with the homogeneous coordinates:
$$[b,c,-1,a]\cdot \begin{bmatrix} R_{1,1}&R_{1,2}&R_{1,2}&0\\ R_{2,1}&R_{2,2}&R_{2,2}&0\\ R_{3,1}&R_{3,2}&R_{3,2}&0\\ 0&0&0&1 \end{bmatrix} \cdot\begin{bmatrix}X_W\\Y_W\\Z_W\\1\end{bmatrix}=0$$
You can treat this matrix times column vector on the right as a transformation of your points, from world coordinates to instrument coordinates. But you can also treat the row vector times matrix on the left as a transformation of your plane, which turns a plane expressed in instrument coordinates into a corresponding plane expressed in terms of world coordinates.
The transformed row vector likely won't have a $-1$ in the third column. But multiples of a vector represent the same plane, so you can just divide all vector elements of the result by the negated third entry to get back to a form that has a $-1$ there.
Note to the casual reader: typically for a plane $p$ (row vector) and a point $q$ (column vector) in homogeneous coordinates, you start with $p\cdot q=0$ as the equation. If you then transform points with a transformation matrix $M$, you get point $M\cdot q$ lying on plane $p\cdot M^{-1}$ because $p\cdot M^{-1}\cdot M\cdot q=p\cdot q$ so one is zero if and only if the other is zero as well. My point here is that usually to perform the same geometric transformation to a plane as you apply to its points, you use the inverse matrix on the coordinate level. We don't see that inversion above because the role of $R$ is reversed in the two use cases. On the one hand it is converting points from world coordinates to instrument coordinates, and on the other hand it is converting instrument plane to world plane. Due to that opposite direction, no inversion is needed, but consider your own application carefully when you decide whether you need inversion.