There is a simple formula for obtaining the 'normal' vector N that is perpendicular to two vectors A and B (that define a corner) in 3D: divide the cross product of A and B with its magnitude.
I'm not good at vector math and haven't seen a solution for obtaining B from A and N. All I've been able to do is figure out a relatively convoluted solution based on Roll-Pitch-Yaw. What is the best standard method of solving this?
Just to be clear, I have A, N and an angle from A that would get me to B.
On
Let's first put aside some technical details like magnitudes and normalization and all that. You basically have A, N, and the angle between A and B. In other words, you have $\vec{a}$, $\vec{a}\times \vec{b}=\vec{n}, \vec{a}\cdot \vec{b}=d$ (say), and you want to find $\vec{b}$. This is what your situation essentially boils down to.
Note that $\vec{a}\times\vec{b}=\begin{pmatrix}a_1\\a_2\\a_3\end{pmatrix}\times\begin{pmatrix}b_1\\b_2\\b_3\end{pmatrix}=\begin{pmatrix}a_2b_3-a_3b_2\\a_3b_1-a_1b_3\\a_1b_2-a_2b_1\end{pmatrix}=\begin{pmatrix}0&-a_3&a_2\\a_3&0&-a_1\\-a_2&a_1&0\end{pmatrix}\begin{pmatrix}b_1\\b_2\\b_3\end{pmatrix}=\begin{pmatrix}n_1\\n_2\\n_3\end{pmatrix}\\=\vec{\vec{A}}\vec{b}=\vec{n}$,
where $\vec{\vec{A}}$ is that antisymmetric matrix that we made using components of $\vec{a}$ (and $\vec{n}$ is known to us). The reason we can't find $\vec{b}$ right away here is because $\det|\vec{\vec{A}}|=0$, so it's non-invertible. There is no unique solution, there are infinitely many $\vec{b}$'s that satisfy this equation.
However, we also have some extra information in the form $\vec{a}\cdot\vec{b}=d$, or, $a_1b_1+a_2b_2+a_3b_3=d$, which gives us enough to be able to pin down $\vec{b}$ uniquely. This condition can be expressed in matrix form as $\vec{\vec{C}}\vec{b}=\vec{d}$, where
$\vec{\vec{C}}=\begin{pmatrix}a_1&a_2&a_3\\a_1&a_2&a_3\\a_1&a_2&a_3\end{pmatrix},~\vec{d}=\begin{pmatrix}d\\d\\d\end{pmatrix}$
Now, we can combine our two pieces of info (matrix equations) together! Let's try adding them, plain and simple: $(\vec{\vec{A}}+\vec{\vec{C}})\,\vec{b}=(\vec{n}+\vec{d})$
Since $\vec{\vec{A}}+\vec{\vec{C}}$ is invertible (you can verify that its determinant is nonzero), this equation can be solved the usual way: $\vec{b}=(\vec{\vec{A}}+\vec{\vec{C}})^{-1}(\vec{n}+\vec{d})$, which expands to $\begin{pmatrix}b_1\\b_2\\b_3\end{pmatrix}=\begin{pmatrix}a_1&a_2-a_3&a_2+a_3\\a_3+a_1&a_2&a_3-a_1\\a_1-a_2&a_1+a_2&a_3\end{pmatrix}^{-1}\begin{pmatrix}n_1+d\\n_2+d\\n_3+d\end{pmatrix}$
Ta-daa!
On
You wrote that you normalize $\vec n$, but you can’t reconstruct $\vec b$ if you do that because then you no longer have any information about the magnitude of $\vec b$. I’ll go with Raad Shaikh and assume that you have $\vec a\times\vec b=\vec n$ and $\vec a\cdot\vec b=d$.
Since the vector $\vec b$ you want is orthogonal to $\vec n$, it’s some linear combination of $\vec a$ and $\vec n\times\vec a$:
$$ \vec b=\lambda\vec a+\mu\vec n\times \vec a\;. $$
You can determine $\lambda$ and $\mu$ by forming the scalar and vector products, respectively, with $\vec a$:
$$\vec a\cdot\vec b=\vec a\cdot\left(\lambda\vec a+\mu\vec n\times \vec a\right)=\lambda|\vec a|^2\stackrel!=d$$
and
$$\vec a\times\vec b=\vec a\times\left(\lambda\vec a+\mu\vec n\times \vec a\right)=\mu\vec a\times\left(\vec n\times \vec a\right)=\mu|\vec a|^2\vec n\stackrel!=\vec n$$
(see vector triple product). Thus
$$\lambda=\frac d{|\vec a|^2}\;,\;\;\mu=\frac1{|\vec a|^2}\;,$$
and
$$ \vec b=\frac{d\vec a+\vec n\times\vec a}{|\vec a|^2}\;. $$
I had a hunch I was missing something simple, and after further study and research found it. The core of the solution is the equation for B:
$$\vec B = \sin (angle) \vec V + \cos(angle) \vec U$$
We obtain U by normalizing A, and V as cross product of N and U. B will be anchored in the same starting point as A and will have a magnitude of 1.