I was at a family reunion yesterday which required a bit of travel. Most of that part of the family lives near one another, so I am the outlier. I can't reasonably expect them to have the next reunion halfway between the cluster and myself because of that fact.
Suppose I were to plot everyone's residence on a map and drop pins at their respective locations. How could I use the latitude and longitude to find the most fair meeting location for the next reunion? Given a cluster of points close together, the logical meeting place would be in the center. But given an outlier, that would pull the meeting place closer to the outlier.
Can I just average the x and y coordinates?
First, it's important that the lat/lons all be fairly similar, or averaging doesn't make sense. Think of two points, one on each side of the international dateline, near the equator. Their lat/lon average is nearly antipodal to both points.
Further (as that example shows), you really want to have points far from the dateline. For real data, that's almost certainly the case, unless your family is fairly unusual, and lives on or near some pacific islands, or is perhaps asian/american, with half the family in Japan and half in Iowa City or something.
Anyhow, with those assumptions, lat/lon averaging isn't crazy...but it's also not ideal, because one degree of latitude is alwa7s 60 nautical miles, but one degree of longitude can be 60 miles (at the equator) or 0 miles (at the poles). So a good first step is to work out an average latitude, say $L$, and multiply all longitudes by $1/cos(L)$. These new coordinates -- call them latitude and pongitude, just for fun -- can then be more reasonably averaged (or have other geometric operations applied).
The reasonable geometric notion of averaging is to say that you want a point $P$ such that the sum of the SQUARED distances from $P$ to all the locations is as small as possible; this is known as a "least squares" problem, and when you solve it, you'll get a latitude and pongitude for $P$, which you can then convert back into a latitude and longitude by multiplying $P$'s pongitude by $cos(L)$.
You should then repeat the process, using the latitude of $P$ as your reference $L$, and continue doing this until things converge to within a mile or two.
A rather different approach is to convert everything to $xyz$ coordinates, using \begin{align} x &= \cos(lon) \cos(lat)\\ y &= \sin(lon) \cos(lat) \\ z &= \sin(lat) \end{align} You can then compute the mean $(x_0, y_0, z_0)$ of these points, either by direct averaging or by least-squares; the result will be a point with $$ r^2 = x_0^2 + y_0^2 + z_0^2 \le 1, $$ and by dividing through by $r$, you get a point $$ (X_0, Y_0, Z_0) = (x_0/r, y_0/r, z_0/ r) $$ on the Earth's surface, from which you can compute the final lat and lon via $$ lat = \arcsin(Z_0);\\ lon = atan2(Y_0, X_0) $$