The Question
What I want to know is how to calculate the mean square distance between a fixed point which is any random point in the sphere and rest of the points.
My Understanding
I'm clueless on how to calculate it. I'm a beginner. First I started from the straight line equation between the two points then equate it to radius of the sphere. then we can take another point. but this is not gonna help right? I think I should use integration here..
Why it is an useful quantity to calculate?
For example, in calculating magnetic moment of an electron around the nucleus the mean squared radius is used.
Hint:
To compute a mean square distance of a point distribution from a given point $(x_0,y_0,z_0)$ you need to compute the integral: $$ \langle r^2\rangle=\frac{\int_V[(x-x_0)^2+(y-y_0)^2+(z-z_0)^2]\rho(x,y,z)\,dV}{\int_V\rho(x,y,z)\,dV}.\tag1 $$ where $\rho(x,y,z)$ is the distribution density. In your example: $$ \rho(x,y,z)\sim\delta(x^2+y^2+z^2-R^2) $$ and $$x_0=(0,0,R), $$ where the origin of coordinates is assumed to be at the center of the sphere.
It remains only to substitute the expressions into (1) and evaluate the integral in spherical coordinates ($x=R\sin\theta\cos\phi,\,y=R\sin\theta\sin\phi, z=R\cos\theta)$.
Can you take it from here?