apologies in advance as I'm a software engineer and not a mathematician so I will probably explain this very poorly. Thanks for muddling through it.
I have a system with three points in three-dimensional space. Specifically the Hip, Knee, and Ankle of a human body. (see the picture)
I can calculate the angle that these three points make at the knee using the Law of Cosines. Check.
However, I need to apply a sign to this angle to indicate whether the knee joint is flexed or hyper-extended. If the Ankle point is "above" the line created by the Knee and Hip it is hyper-extended. If "below", it is flexed.
This is complicated to me because the knee doesn't seem to bend along the plane that the three points create (highlighted by the following image which shows the angle between the femur and tibia/fibula). In addition, flex and hyper-extend of the knee is relative to the position of the knee, and doesn't neatly fall along any of the axis. (i.e. I can't compare the x-position of ankle > knee)
So, how can I determine if the Ankle point is "above" or "below" the line created by the Knee and Hip? (If there is a way to calculate the signed angle then I'll take that, but I'm assuming this will need to be calculated separately.)
I thought of calculating the unit vectors of Hip-->Knee and Knee-->Ankle but I'm not sure how to reliably compare them.
I also thought of re-orienting the whole coordinate system so that the Knee is (0,0,0) and the hip is (+x,0,0) and seeing if the ankle is (0,±y,0)... but I've no idea how to do that either.
Thanks,


Here I am proposing a way to re-orient the whole coordinate system as described in your last paragraph.
Let the knee be the origin and the hip $H$ be at $(x_H,0,z_H)$, $x_H\ne0$. $xy$-plane is horizontal and $z$-axis is vertical.
Let $T$ be a shear transformation that translates points vertically depending on their $x$-coordinates, and translate the hip point onto $x$-axis. Then $T$ transforms the knee-hip line to the $x$-axis.
$T$ transforms the point $(1,0,0)$ by translating it by $\frac{z_H}{x_H}$ towards negative $z$-direction:
$$T(1,0,0) = \left(1, 0, -\frac{z_H}{x_H}\right)$$
$T$ does not change the points $(0,1,0)$ and $(0,0,1)$. So $T$ can be represented by the matrix
$$T\pmatrix{x\\y\\z} = \pmatrix{1&0&0\\0&1&0\\-\frac{z_H}{x_H}&0&1}\pmatrix{x\\y\\z}$$
Perform the same transformation on the ankle point, and see if the transformed result has positive (above) or negative (below) $z$-coordinate.