EDIT: I have solved the problem for my particular distance function using alternative ways however I think it's probably worth it to keep this question open as it would be interesting to find a more generalized formula for closest point in a line segment for any distance function. After all my, question still stands.
I'm trying to find the distance from a line segment AB to a point C in 2D space. This is a pretty normal problem solved using the formula
On the numerator we have twice the area of the triangle ABC and on the denominator we have the distance AB. Now this formula gives the distance from C to the point that is closest to the line that passes in A and B. The way to solve this and to have the closest point only in the line segment is to simply limit the values to be between the distance from C to A and B, this way if the closest point is outside of the line segment we will instead get the distance from C to A or to B, whichever is closest. This works obviously because A and B are the closest and farthest distances to C, at least when we consider the Euclidean distance.
My problem is what if I'm using a different distance function that isn't based just on the coordinates of the point so we might have a situation where the closest point is actually in the middle of the line segment. I can no longer limit the values to be between the distances to A and B. This image should help see the problem.
I want to find the distance from C to the line segment AB. Obviously I had to adapt the previous formula to my distance function but that isn't a problem and I know it's working correctly. In this example the actual closest point in the segment is D. It's closer than A and B. However it calculates that E is the closest. If this was in Euclidean distance I could just limit it to be between A and B but it's not an option here. Any ideas or hints about what to do?