I looking for method which can compute the yellow area in this image..
The ball with the green fill is a sphere, where i know the center point and the radius of it. The circle with the red fill outside i a point outside the sphere.
How do find the point which is closest to the points, and some points around around the closest point (indicated by the yellow marking)?

Let's say your sphere has radius $R$ and center $(a,b,c)$. Your point is $(A,B,C)$.
The distance $D$ from the point to the center of the sphere is
$$D = \sqrt{(A-a)^2 + (B-b)^2 + (C-c)^2}.$$
A unit vector from the center of the sphere, toward your point, is
$$\left(\frac{A-a}{D}, \frac{B-b}{D}, \frac{C-c}{D}\right).$$
Then, to reach the point on the sphere closest to your point, start at the center, and go in the direction of the vector a distance $R$:
$$P = \left(a + R\left(\frac{A-a}{D}\right), b + R\left(\frac{B-b}{D}\right), c + R \left(\frac{C-c}{D}\right)\right).$$