I'm a programmer and I'm doing a whitebalance-transformation in RGB colorspace. This should work with this transformation matrix that I've found in literature:
$$ \begin{pmatrix} R \\ G \\ B \end{pmatrix} = \begin{pmatrix} \frac{255}{R_w} & 0 & 0 \\ 0 & \frac{255}{G_w} & 0 \\ 0 & 0 & \frac{255}{B_w} \end{pmatrix} \times \begin{pmatrix} R´ \\ G´ \\ B´ \end{pmatrix} $$ With a colorspace where the range of all possible values is $ \Big[0, 255\Big]$ (which is 8 bit).
I've been reading that a cross-product including matrices (as seen above) can be solved like a system of linear equations.
$$ R = \big( \frac{255}{R_w} + 0 + 0 \big) \cdot R´ = \frac{255}{R_w} \cdot R´ \\ G = \big( 0 + \frac{255}{G_w} + 0 \big) \cdot G´ = \frac{255}{G_w} \cdot G´ \\ B = \big( 0 + 0 + \frac{255}{B_w} \big) \cdot B´ = \frac{255}{B_w} \cdot B´ \\ $$
But that seems to be a faulty approach as $R$, $G$ and $B$ can be bigger than 255 which is no valid result of that transformation.
Example
Original pixel: $$ R´ = 111; G´ = 154; B´ = 255 $$
Whitespace transformation input: $$ R_w = 123; G_w = 138; B_w = 217 $$
Results with my approach: $$ R = \frac{255}{123} \cdot 111 = 230 \\ G = \frac{255}{138} \cdot 154 = 284 \\ B = \frac{255}{217} \cdot 255 = 299$$
Obviously, the results are out of the range of my 8 bit colorspace.
How to correctly solve this whitespace transformtation? I'm mainly lost as I have no idea how to handle the matrix within a cross-product.
A cross product is a product between two vectors. In your case, the symbol $\times$ doesn't denote a cross product, just an ordinary multiplication of a matrix and a vector. Since the matrix is diagonal, the multiplication can indeed be reduced to the equations $R=(255/R_w)R'$ and likewise for $G$ and $B$ (though the additional zeros you include in parentheses don't make sense the way you wrote them). Thus your problem isn't a problem with vectors and matrices; you need to figure out how you obtained colour values greater than the white balance parameters (or equivalently white balance parameters below the greatest colour values).