vector $\tilde a = c\tilde i+d\tilde j+e\tilde k$ is our input, and vector $\tilde b = f\tilde i+g\tilde j+h\tilde k$ is our output, and $\theta$ is the angle to rotate by. Essentially $\tilde b = f(\tilde a, \theta)$
This is similar to Components of a 3d vector given specific angles, but I need to rotate the vector on the plane created by $\tilde a$ and $\tilde j$.
I attempted to create a function by converting $\tilde a$ to a 2d point on this plane:
$(x, y) = (\sqrt{1-d^2}, d)$
converting to a polar coordinate, adding alpha, converting back to cartesian, and taking the $y$ component as the value for $g$:
$$g = \sqrt{d^2+\sqrt{1-d^2}} \times \sin\bigg(\tan^{-1}\Big(\frac{d}{\sqrt{1-d^2}}\Big)\bigg)$$ $$f = \frac {c \times\sqrt{1-d^2}} {\sqrt{c^2+e^2}}$$ $$h = \frac {e\times f}{c}$$
This formula produces unexpected results when given a value close to $\pm\tilde j$, eg. $f(0.174126\tilde i+0.984723\tilde j, \frac {-\pi}{16})$ returns $g = 0.996584$ which is higher the input value of $d=0.984723$. The expected value is aproximately $g=0.779171$ (I think). Continuing to tilt the resulting $\tilde b$ by $\frac{-\pi}{16}$, cases $g$ to aproach $0.996$ ish.
Can anyone spot my error, or find a better way of doing this?