I need to calculate the resultant velocities of 3 circles/masses/particles if they was to collide at the exact same time. I understand that this is theoretically impossible (or incredibly unlikely) to happen, but that's beyond the point.
Here is an example of what I mean

The blue line represents the velocity of the blue ball, the blue ball's X coordinate is exactly between the other two balls. The collisions are elastic.
I have asked my mathematics teachers about this problem and all of them said that it was impossible, since when you use conservation of momentum formula's you have too many unknowns. That being said, is there something that would give an accurate approximation?
Preferably, the radius' and mass's and velocities are variable, but I would be happy if they was kept constant if it means there is a formula (With the exception of the blue circles velocity).
Thank you.
The reason the answer is indeterminate (without some further specification of the nature of the collisions) is that you have (assuming everything is in a plane) more unknowns than equations.
For example, if any case were solvable, it would be the case where
In that case, the there are three parameters describing the answer (the final Y velocity component of the blue ball, the (equal) final Y velocities of the red balls, and the final X velocity of one of the red ball -- the other one gets the negative of this velocity component). And there are naively three equations: conservation of energy, and conservation of X and Y momenta. But the conservation of X momentum is automatic by the symmetry imposed, so there are really only two equations for three unknowns.
Now, say you know something else (for example, that the red balls leave the collision at an opening angle of 120 degrees). Then that third constraint allows you to easily solve the equations and determine the respective velocities.
The way a "frictionless billiards" simulation would work is to model the repulsion force between each pair of balls as some sort of rapidly increasing conservative force with a (fuzzy, but not very fuzzy) force radius of $R$. Then you could take the limit as the potential looks more and more like a step function. You would find (ignoring introduction of spin because there is no friction) that for all three balls identical the opening angle would in fact be 120 degrees, and the solution is that the blue ball continues on with 1/3 of its original velocity, while the two red balls get a speed of twice that.
Physically, the rotation of the two red balls can't be ignored, so your simple billiards simulation would not be that accurate.