Why do I get a domain error for the following computation using law of cosines

1.6k Views Asked by At

Following are two points. lat1, lon1 represent one point and lat2, lon2 represent another point.

lat1, lon1 = 37.317233, -121.94331
lat2, lon2 = 37.317233, -121.94331

When I use the following formula (law of cosines) I get a math domain error

sine_part = sin(lat1_in_radians) * sin(lat2_in_radians)
distance = RADIUS_OF_EARTH_IN_METRES * acos(sine_part + cos(lat1_in_radians) * cos(lat2_in_radians) * cos(lon2_in_radians - lon1_in_radians))

where degrees to radians is converted by decimal_degree / 57.2957795

The weird thing is, the following points work as expected though. I get 0 as my distance.

lat1 = 54.11102, lon1 = -111.12345
lat2 = 54.11102, lon2 = -111.12345

I do understand that the values resulted by applying one of these trignometric functions is not in the interval it could work with. However I would like to know why this error happens in more detail from one of the experts here please.

I'm hoping I haven't done a school boy error, however I have forgotten my school maths so I won't be surprised if I have missed a trivial point!

1

There are 1 best solutions below

0
On

This should fix your code:

sine_part = sin(lat1_in_radians) * sin(lat2_in_radians)
temp = sine_part + cos(lat1_in_radians) * cos(lat2_in_radians) * cos(lon2_in_radians - lon1_in_radians)
if temp > 1:
    temp = 1
if temp < -1:
    temp = -1
distance = RADIUS_OF_EARTH_IN_METRES * acos(temp)

As everyone is telling you in the comments, because of floating point errors, temp, which should be between $-1$ and $1$, might be fractionally outside of this range.

(By the way, I don't know what programming language you are using. But I think the code I wrote is self explanatory.)