As an input I have two colors, let's say red (RGB = 1,0,0) and magenta (RGB = 1,0,1).
Now I have an image which includes additive mixes between these two colors, for example purple (RGB = 0.5,0,1).
I want to calculate the mix amount between these two colors where 0 is 100% the first color (red) and 1 100% the second color (magenta). In the example with purple, the mix would be 0.5.
Of course input colors may be as complicated as possible, but it is always ensured that they are indeed mixable.
I know how to calculate this for grayscale colors, but not for arbitrary input colors.
A visualized input and output would be like this, when the output will be used as the alpha channel for the second color:

Things became much easier to me when I thought of the RGB components as XYZ coordinates of a 3-dimensional point, and that the lengths between those is the key to success!
To get the mix-amount between two colors, you require some basic vector maths:
The resulting a is exactly 0 if the color is completely the background color A. It is 1 if the color is completely foreground color B. Anything between is background color A + foreground color B * a.
I'm sorry if any of my math formatting is incorrect, I'm new to LaTeX and not really a mathematician :)