I'm trying to parametrize the boundary of the u'v' color space (CIE1976; see image below)*. I know the transformations to go from RGB color space, and I know the boundaries of the RGB space. How does one calculate the boundary of the transformed space? The space is,
$$ u' = \frac{4X}{X + 15Y + 3Z} \\ v' = \frac{9Y}{X + 15Y + 3Z} $$
Where, \begin{equation*} \begin{pmatrix} X \\ Y \\ Z \\ \end{pmatrix} = \begin{pmatrix} 0.49 & 0.31 & 0.20 \\ 0.17697 & 0.8124 & 0.01063 \\ 0.0 & 0.01 & 0.99 \end{pmatrix} \begin{pmatrix} R \\ G \\ B \end{pmatrix} \end{equation*}
And: $R,G,B\in[0.0, 1.0]$
*Really, I want the boundary of u'v' space, but parametrized in terms of RGB... but I think I can handle the reverse transformations to get that part.
For example, for red:
RGB $ = [1, 0, 0]$
XYZ $ = [0.49, 0.1769, 0.0]$
u',v' $ = [0.623, 0.506]$
which is the right-most point on the triangle.
The image of the unit RGB cube under this mapping is a triangle that falls within the pictured area. As Rahul points out in his comment, the RGB gamut doesn’t cover the entire visible spectrum. To generate the boundary of the pictured area, you need other data besides the RGB-to-u'v' transformation.
It’s not hard to see that the image of the unit RGB cube is a triangular region in u'v' space. The XYZ-to-u'v' transformation is a perspectivity that maps lines through the origin (less the origin itself) in XYZ space to a single point in u'v' space. The RGB-to-XYZ mapping transforms the unit cube into a paralellepiped with a vertex at the origin and planes through the origin in XYZ space are mapped to lines in u'v' space so the boundary of the image is a triangle.
Indeed, writing the RGB-to-u'v' as $$\mathbf p_{u'v'} = \begin{bmatrix}4&0&0\\0&9&0\\1&15&3\end{bmatrix} \begin{bmatrix} 0.49 & 0.31 & 0.20 & 0 \\ 0.17697 & 0.8124 & 0.01063 & 0 \\ 0.0 & 0.01 & 0.99 & 0\end{bmatrix}\begin{bmatrix}R\\G\\B\\1\end{bmatrix} = \mathtt A \left[\begin{array}{c|c} \mathtt M & \mathbf 0 \end{array}\right] \begin{bmatrix}R\\G\\B\\1\end{bmatrix}$$ followed with a dehomogenization of $\mathbf p_{u'v'}$ by dividing through by its last component, makes it recognizable as a pretty standard pinhole camera projection with the camera at the origin. The linear part $\mathtt{AM}$ of this projection could be refactored into the product $\mathtt{UR}$ of an upper-triangular and orthogonal matrix and $\mathtt U$ normalized for a more conventional factorization of this projection into intrinsic and extrinsic matrices.