I tried searching the internet but I didn't find a definite answer.
Barycentric coordinates can be used for all sort of stuff however the formula I know to calculate them can potentially cause a division by 0 when 2 (and thus 3) of the edges of a triangle are colinear.
This is how I would calculate the barycentric coordinates:
Eigen::Vector3f v0 = b - a, v1 = c - a, v2 = p - a;
float d00 = v0.dot(v0);
float d01 = v0.dot(v1);
float d11 = v1.dot(v1);
float d20 = v2.dot(v0);
float d21 = v2.dot(v1);
float denom = d00 * d11 - d01 * d01;
float v = (d11 * d20 - d01 * d21) / denom;
float w = (d00 * d21 - d01 * d20) / denom;
float u = 1.0f - v - w;
return {u, v, w};
The question is then, what could be an elegant way to find the barycentric coordinates of a degenerate triangle? Trivially it should be lerping between the 2 farthest end points, but I want to do this mathematically, not by adding if statements and checking here and there.
I don't think there's a mathematical way of doing what you want, at least not one that does not have the equivalent of if statements. As a commenter has pointed out, barycentric coordinates for a degenerate triangle will not work outside of a line or point. Even where they work they will not be unique. If your algorithms are unduly sensitive to degenerate triangles I'd suggest removing them in a preprocessing step, or finding an algorithm that is less sensitive to the pathologies of triangle soup. There's lots of literature on both approaches.