OpenCv uses the following polynomial for estimating lens distortion. More information can be found here.
$\frac{1+k_1r^2+k_2r^4+k_3r^6}{1+k_4r^2+k_5r^4+k_6r^6}$
There is a lot more to the calculation than just that but I think for the question that is all that is needed. When OpenCv is calculating the approximation it uses a bundle adjustment which I believe to be the Levenberg/Marquardt. OpenCv in general does well with regular lens, but fisheye lens seems to be problematic. The main issue is coverage of the calibration points. If you do not fully cover the image with calibration points the LM adjustment will (as expected) produce a rather nice estimation for the area covered by the calibration points but get rather radical in areas where no points are.
So onto my question, is there a mathematical way to provide an error/constraint into the Hessian matrix that would take the curvature of the approximation into account as so to penalize radical approximations?