I am given 2 points, each point has an associated direction vector. One point and direction vector pair represents the upper leg (femur) and the other point and direction vector pair represents the lower leg (tibia). The pairs represent the bones as seen from the front.
Now I am asked to calculate the flexion / extension angle (rotation of the vectors around the x axis of the knee), or the movement of the leg in and out of the image.
I am also asked to calculate the varus / valgus angle between the femur and tibia. This represent the movement / sliding of the femur / tibia along the x axis.
I tried to draw it. P1 and V1 represent the femur, P2 and V2 represent the tibia. A1 represent the flexion / extension angle, A2 represent the varus / valgus angle.
Let me change your notation of the axes in order to keep the standard chirality.
So let's take the pelvis to lie on the $y,z$ plane, with the origin at the hip joint $H$ and with the $x$ axis in the retro-antero direction.
In standing up position the femur will extend for a length $f$ along the negative $z$ axis and so the knee $K$ will be at $(0,0,-f)$.
We take the reference system $x',y',z'$ to be fixed with the femur, with origin in $K$ and being parallel to $x,y,z$ when standing up.
We can model the knee joint as a hinge with a single degree of freedom.
Apart from the varus angle $\beta$, the hip is also capable of a moderate "torsion" angle $\alpha$, which corresponds to rotating the foot in-out ( I don't know the anathomical term for that).
We take $\beta$ to be a right-hand rotation around $x'$, moving $z'$ to $z''$, and $\alpha$ to be a right hand-rotation around $z'$, (not $z''$). We have then the flexion, that we individuate with $\phi$, the right-hand angle around $y''$, which theoretically spans from$0$ to $\pi$.
The tibia will extend a length of $t$ along $-z''$. from the knee $K$ to the ankle $A$.
So the reference system $x'',y'',z''$ is placed with the origin in $K$, is fixed with the tibia extending along $-z''$ to $(0,0,-t)$ and with the axis $y''$ being the axis of the knee joint.
We pass then to define the movement of the knee in the frame fixed with the pelvis.
We have two possible angular movements, which we choose to define as:
- a rotation $\theta$ around $x$, which defines the lateral inflexion;
- a rotation $\eta$ around $y$, which defines the frontal inflexion, with positive values corresponding to a backward flexion;
Having defined the kinematic model, we pass now to the matematical analysis.
The matrices $$ {\bf R}_{\,{\bf x}} (\alpha ) = \left( {\matrix{ 1 & 0 & 0 \cr 0 & {\cos \alpha } & { - \sin \alpha } \cr 0 & {\sin \alpha } & {\cos \alpha } \cr } } \right)\quad {\bf R}_{\,{\bf y}} (\beta ) = \left( {\matrix{ {\cos \beta } & 0 & {\sin \beta } \cr 0 & 1 & 0 \cr { - \sin \beta } & 0 & {\cos \beta } \cr } } \right)\quad {\bf R}_{\,{\bf z}} (\gamma ) = \left( {\matrix{ {\cos \gamma } & { - \sin \gamma } & 0 \cr {\sin \gamma } & {\cos \gamma } & 0 \cr 0 & 0 & 1 \cr } } \right) $$ represent the unit column vectors of a reference system, rotated wrt to the base system, around the axis indicated and for an angle measured according to the right-hand rule.
Formally, the expression $$ {\bf v'}_b = {\bf R}_{\,{\bf x}} (\alpha )\;{\bf v}_b $$ returns the vector $\bf {v'}$ which is the vector$\bf v$ rotated by the angle $\alpha$ around the $x$ axis.
Both the vectors are column vectors expressed in the same base system as the axis $x$: that's what the suffix $b$ is going to remind.
Therefore, taking the unit vectors along the axes $ {\bf i} , {\bf j} , {\bf k}$, joined horizontally into a matrix, the above translates into $$ {\bf U}'_b = \left( {\matrix{ {{\bf i}'} & {{\bf j}'} & {{\bf k}'} \cr } } \right)_b = {\bf R}\,\;\left( {\matrix{ {\bf i} & {\bf j} & {\bf k} \cr } } \right)_b = {\bf R}\,{\bf U}_b = {\bf R}\,{\bf I} = {\bf R} $$
Regarding the composition of the rotations, we have two possible schemes: extrinsic and intrinsic definition.
If we apply a rotation around $x$, followed by a rotation around the $y$ axis, the last being (as the first) relevant to the base system, then we have the extrinsic definition, and the global rotation, in the base system, will be expressed by the right-to-left product of the matrices $$ {\bf R}_{\,{\bf T}} = {\bf R}_{\,{\bf y}} \;{\bf R}_{\,{\bf x}} $$ If instead, the rotation about $x$ is followed by a rotation around the "new" $y'$ axis (intrinsic definition), since in the base system this is expressed as $$ {\bf R}_{\,{\bf T}} = {\bf R}_{\,{\bf y}'} \;{\bf R}_{\,{\bf x}} = {\bf R}_{\,{\bf x}} \;{\bf R}_{\,{\bf y}} \;{\bf R}_{\,{\bf x}} ^{\, - \,1} \;{\bf R}_{\,{\bf x}} = {\bf R}_{\,{\bf x}} \;{\bf R}_{\,{\bf y}} $$ that is with the product left-to-right of the corresponding matrices.
That premised, we can conclude
a) concerning the vector $\vec{HK}$, applying first $\eta$ then $\theta$, both extrinsic $$ \eqalign{ & {\bf U}'_b = {\bf R}_{\,{\bf x}} (\theta )\;{\bf R}_{\,{\bf y}} (\eta ) \cr & \vec {HK} = {\bf U}'_b \left( {\matrix{ 0 \cr 0 \cr { - f} \cr } } \right) = - f\;{\bf R}_{\,{\bf x}} (\theta )\;{\bf R}_{\,{\bf y}} (\eta )\;{\bf k} \cr} $$
b) concerning the vector $\vec {KA}$, applying first $\beta$, then $\alpha$ extrinsic, then $\phi$ intrinsic, in the reference $x',y',z'$ $$ \eqalign{ & {\bf U}''_b = {\bf R}_{\,{\bf z}} (\alpha )\;{\bf R}_{\,{\bf x}} (\beta )\;{\bf R}_{\,{\bf y}} (\phi ){\bf U}'_b = \cr & = {\bf R}_{\,{\bf z}} (\alpha )\;{\bf R}_{\,{\bf x}} (\beta )\;{\bf R}_{\,{\bf y}} (\phi )\;{\bf R}_{\,{\bf x}} (\theta )\;{\bf R}_{\,{\bf y}} (\eta ) \cr & \vec {KA} = {\bf U}''_b \left( {\matrix{ 0 \cr 0 \cr { - t} \cr } } \right) = - t\;{\bf R}_{\,{\bf z}} (\alpha )\;{\bf R}_{\,{\bf x}} (\beta )\;{\bf R}_{\,{\bf y}} (\phi )\;{\bf R}_{\,{\bf x}} (\theta )\;{\bf R}_{\,{\bf y}} (\eta )\;{\bf k} \cr} $$
c) and finally, we have of course $$ \eqalign{ & \vec {HK} = \vec {HK} + \vec {KA} = \cr & = - \left( {f\;{\bf I}\; + t\;{\bf R}_{\,{\bf z}} (\alpha )\;{\bf R}_{\,{\bf x}} (\beta )\;{\bf R}_{\,{\bf y}} (\phi )} \right){\bf R}_{\,{\bf x}} (\theta )\;{\bf R}_{\,{\bf y}} (\eta )\;{\bf k} \cr} $$