The question is quite hard to explain so I included a diagram, what I'm trying to do is to solve for the two angles indicated by clouds, in the quickest way possible. As for the vectors, they are 3 perpendicular vectors, and they are all unit vectors, they form an orthonormal basis!
My first instinct was to solve for a 3x3 linear system using Cramer's rule, such that:
$$(x, y, z) = a * \vec{fwd} + b * \vec{up} + c * \vec{rght}$$ And then just use a, b, c to trigonometrically extract the angles. This works just fine, but it so expensive... Another alternative I am looking at are spherical coordinates, but I have no idea how to make these work ...
I'd favor an approach that is atleast faster than just solving a 3x3 system, as this is a bottleneck in my application, especially when checking thousands vectors per second.
Since your reference direction vectors are an orthonormal basis, you just need to rotate (and perhaps translate) into the “rfu” coordinate system. This is the inverse of the transformation that maps the standard basis to this one and the inverse of a rotation matrix is its transpose, so the required rotation matrix has $\vec r$, $\vec f$ and $\vec u$, in that order, as its rows. Therefore, the transformed coordinates of a point $\vec p$ are $$a = \vec r\cdot\vec p \\ b = \vec f\cdot\vec p \\ c = \vec u\cdot\vec p.$$ (In general, the coordinate of a point is the scalar projection onto a basis vector in a direction parallel to the span of the other basis vectors.) The two angles are then $\arctan\frac if$ and $\arctan\frac rf$. Presumably you’ve got access to an
ATAN2-type function that’ll give you the correct quadrant for the computed angle.