I need to check this formula I have to compute the distance of a point with latitude $lat$ from the equator:
$$ \mathrm{feet} = \mathrm{lat} * 10000 \times 3280 / 90 $$
Example: A point at $40.480125^\circ N$ has distance
40.480125 * (10000 / 90) * 3280 = 14752756.66666667 ft
from the Equator. My stored value for this is 1417623.
The problem I am having is the numbers are not making sense. I am told I have data that was already converted and is a 7 digit number, but all my calculations end up being an 8 digit number with a decimal.
Why doesn't the result make sense ?
The formula is taking the circumference of the Earth (assumed to be a perfect sphere) as exactly 40,000 km., which is OK to two significant figures. The arclength along the Earth's surface is then found as
$$ \frac{\text{angle from Equator}}{90º} \ \cdot \ 40,000 \ \text{km.} \ \ , $$
and the result is converted to feet using $ \ \frac{3280 \ \text{ft.}}{1 \ \text{km.}} \ $ , alright to three significant figures. The result is good to two significant figures. Your result for 40.48º North should have eight digits, so someone else slipped up...