I'm using the following formula to calculate the new vector positions for each point selected, I loop through each point selected and get the $(X_i,Y_i,Z_i)$ values, I also get the center values of the selection ($X,Y,Z$) , call them $(X_c,Y_c,Z_c)$ The distance of each point from the center is $d_i=\sqrt{(X_i-X_c)^2+(Y_i-Y_c)^2+(Z_i-Z_c)^2}$.
The coordinates for the new vector position is:
$g_i=\left(\frac b{d_i}(X_i-X_c)+X_c,\frac b{d_i}(Y_i-Y_c)+Y_c,\frac b{d_i}(Z_i-Z_c)+Z_c\right)$
My problem is I don't think it's averaging properly, it should be a smooth path or an average path the whole way from every axis.
Here's a screen shot before I run the script:

And here is what happens after:

It's perfect on the front axis, but as you can see from the top and the side it's not so smooth.
I'm using Python in MAYA to calculate this, here's the code I'm using:
import maya.cmds as cmds
import math
sel = cmds.ls(sl=1, fl=1)
averageDistance = 0
cmds.setToolTo('Move')
oldDistanceArray = []
cs = cmds.manipMoveContext("Move", q=1, p=1)
for i in range(0, len(sel), 1):
vts = cmds.xform(sel[i],q=1,ws=1,t=1)
x = cs[0] - vts[0]
y = cs[1] - vts[1]
z = cs[2] - vts[2]
distanceFromCenter = math.sqrt(pow(x,2) + pow(y,2) + pow(z,2))
oldDistanceArray += [(distanceFromCenter)]
averageDistance += distanceFromCenter
if (i == len(sel) -1):
averageDistance /= len(sel)
for j in range(0, len(sel), 1):
vts = cmds.xform(sel[j],q=1,ws=1,t=1)
gx = (((averageDistance / oldDistanceArray[j]) * (vts[0] - cs[0])) + cs[0])
gy = (((averageDistance / oldDistanceArray[j]) * (vts[1] - cs[1])) + cs[1])
gz = (((averageDistance / oldDistanceArray[j]) * (vts[2] - cs[2])) + cs[2])
cmds.move(gx,gy,gz,sel[j])
cmds.refresh()
Aditionally, I have found another 'error' here: (before)

After: 
It should draw a perfect circle, but it seems my algorithm is wrong
In your second example, at least, if you want it to be a circle with the yellow points that I can see highlighted, simply changing the distance to the centre won't be enough, as there are no points along the green axis to be modified to form the perfect circle. However returning to your original problem, the issue to me seems to be that half of the points selected are skewed to the back of the yellow axis, effectively moving your centre to a position that isn't actually the centre you want. By the looks of it, the centre that you want is the geometric centre - given the points you have, you should find $X_c, Y_c$, and $Z_c$ independently by finding the two points furthest from each other along each axis, and then finding the point, along that axis, that lies between those two points