I'm working on a computer vision project where I'm trying to detect a moving colored circle (doughnut actually). After some work I'm able to get it working pretty well except for these two edge cases. My results are always a small set of approximately 5-10 points that describe a circle. Then I can calculate the "center of mass" of the object. When this comes out right like a circle I get the center coordinates correctly. But when it comes out like the sketches below the center point is off.
The first image is the result of a little extra object being detected, and the second case is usually the camera occluding part of the image or sometimes an aggressive light source. A lot of my error cases have part of a circle in them.
So I've been trying to think if there is a neat mathematical way to identify each edge case, and then solve it, by figuring out where the real center would be of a circle that fit to it. I can see how to approach this from a trial and error convergence sort of approach. But I thought there might be something smarter I hadn't thought of.
I do know I could solve the problem on the right with an existing function if I had to, but I don't have a solution for the left. I don't want the circle that surrounds all the points on the left, I'm after what the circle would be if that little nub didn't exits. And I'd love to have a good understanding of solving both edge cases.
Any advice is always appreciated it.

One way to fit points to a circle is to do linear regression on the equation in the form $x^2 + a x + y^2 + b y + c = 0$. Writing this as $(x+a/2)^2 + (y+b/2)^2 = (a^2 + b^2)/4 - c$, this corresponds to a circle of centre $(-a/2, -b/2)$ and radius $\sqrt{(a^2+b^2)/4 - c}$.