While this question is related to astronomy, I believe it's a spherical trigonometry problem as outlined below.
Below are two different sets of equations for computing the Altitude and Azimuth of a given point in Right Ascension and Declincation (e.g. a star). When using the second set of equations, I always get the correct answer (verified with NASA JPL Horizons). When using the first set of equations, I get the right answer for $a$. The second equation in the first set, usually gives the right answer, unless $h$ is negative, and the first equation only occasionally gives the right answer.
Both sets of equations appear similarly in many computational astronomy books. In the book "3D Astronomy with Java", the author derives the first set using spherical trigonometry. In the book "Fundamental Astronomy by Karttunen et al", they state that both the cos and sin of $A_z$ are needed to determine the correct quadrant. And in the "Nautical Almanac" the reduction procedures test if $h$ is negative and subtract it from 90, but that doesn't produce the right answer for me. The equations are as they appear in "The Explanatory Supplement to the Nautical Almanac 3rd Ed".
So, my question is:
How would one use the first set of equations?
$a$ is altitude, $A_z$ is azimuth, $h$ is the hour angle, $\delta$ is the declination, $\phi$ is the latitude. I did modify the second set of equations to use the same variables at first, so there's possibly a transcription error, just know that they are only included for completeness.
First Set:
$ \cos \mathit{a} \sin A_{z} = -\cos \delta \sin \mathit{h} \\ \cos \mathit{a} \cos A_{z} = \sin \delta \cos \phi - \cos \delta \cos \mathit{h} \sin \phi \\ \sin \mathit{a} = \sin \delta \sin \phi + \cos \delta \cos \mathit{h} \cos \phi $
Second set:
$ \tan A_z = \frac{\sin a}{\cos h \sin \phi - \tan \delta \cos \phi} \\ \sin a = \sin \phi \sin \delta + \cos \phi \cos \delta \cos h $
Implementation of the first set:
a=Math.asin(Math.sin(dec)*Math.sin(lat)+Math.cos(dec)*Math.cos(H)*Math.cos(lat));
az=Math.acos((Math.sin(dec)*Math.cos(lat)-Math.cos(dec)*Math.cos(H)*Math.sin(lat))/(Math.cos(a)));
az=Math.asin((-Math.cos(dec)*Math.sin(H))/(Math.cos(a)));
Implementation of the second set:
az = (Math.atan2(Math.sin(H), Math.cos(H)*Math.sin(lat) - Math.tan(dec)*Math.cos(lat)));
a = (Math.asin(Math.sin(lat)*Math.sin(dec) + Math.cos(lat)*Math.cos(dec)*Math.cos(H)));
az-=Math.PI; //This version produces Az measured from the South, subtract 180
Through trial and error, I have figured it out. I have labeled the equations below. The solution depends on the signs of $\cos A_z$ and $\sin A_z$.
if $\cos A_z$ > 0 and $\sin A_z$ > 0: eq1 and eq2 are correct
if $\cos A_z$ > 0 and $\sin A_z$ < 0: eq1 is correct
if $\cos A_z$ < 0 and $\sin A_z$ < 0: $A_z$ = 360 - eq2
if $\cos A_z$ < 0 and $\sin A_z$ > 0: eq2 is correct
$ \textbf eq1 \cos \mathit{a} \sin A_{z} = -\cos \delta \sin \mathit{h} \\ \textbf eq2 \cos \mathit{a} \cos A_{z} = \sin \delta \cos \phi - \cos \delta \cos \mathit{h} \sin \phi \\ \textbf eq3 \sin \mathit{a} = \sin \delta \sin \phi + \cos \delta \cos \mathit{h} \cos \phi $