I have a plane A in 3D which can be defined either by the scalar plane equation or by its normal n. I want to find a new plane B which is orthogonal to plane A but shares an edge with plane A. I.e. I want to rotate plane A 90 degrees along a line defined by one of its edges.
I may be asking how do I use Rodrigues formula to do this but at this point I need some clarification of the process of figuring out what the Rotation matrix is going to be. Any initial suggestions on how to go about this?
TL;DR The easiest way to go is to ask yourself, what points stay fixed under rotation. The answer is that these are exactly the points of the axis of rotation. If the axis of rotation lies in your plane, then surely it's going to be the intersection of this plane and it's rotated copy, provided that the angle of rotation is not a multiple of $\pi$.
Now, I want to fill in the blanks. Let $A$ be a 3-by-3 rotation matrix, that is a matrix satisfying the equation $A^T A = 1$ (where by $1$ I denote the unit matrix) and $\det A = 1$. This matrix has a real eigenvalue $\lambda$: indeed, its characteristic polynomial has degree $3$, and such polynomials always have a real root.
$A^{-1} = A^T$, therefore $A^T$ has an eigenvalue $1/\lambda$. But $A^T$ has the same characteristic polynomial as $A$. If the other two eigenvalues of $A$ are complex, we can immediately conclude that $\lambda = 1$ (and the complex eigenvalues are complex conjugate of length $1$). If they are real, in general we can conclude that $A$ has eigenvalues $\lambda$, $1/\lambda$ and $\mu = \det A = 1$. Take an eigenvector $x$ corresponding to $\lambda$. $x \cdot x = Ax \cdot Ax = \lambda^2 x \cdot x$, therefore $\lambda = \pm 1$.
The case when eigenvalues are all $1$ corresponds to trivial rotation. The case when eigenvalues are $1$, $-1$, $-1$ corresponds to rotation by $\pi$ (because eigenvectors of a rotation matrix corresponding to different real eigenvalues are orthogonal: $x \cdot y = Ax \cdot Ay = - x \cdot y$ can only hold if $x \cdot y = 0$).
The eigenspace of eigenvectors $\mu$ and $1/\mu$ is orthogonal (proven above), and it is easy to see that birestriction of $A$ on it is still a rotation. It is now easy to see that any plane orthogonal to the axis of rotation is preserved by $A$, so a rotation in 3D is indeed rotation around the axis of rotation.