From the book Astronomy, principles and practice. I cannot solve the second part.
Assuming the Earth to be a sphere of radius 6378 km calculate the great circle distance in kilometers between London (51◦ 30 N, 0◦) and New York (40◦ 45 N, 74◦ W). Find also the most northerly latitude reached on the great circle arc.
the great circle distance is given by
$\Delta\sigma=\arccos(\sin51.5\sin40.75 + \cos51.5 \cos40.75\cos74)$
Then I converted the result to radians and used $d=r\Delta\sigma$ to get 5786km. Then I tried to use the departure equation but logically it does not seem to correspond to the problem.
I suppose the formulas you can use are the ones on this page: http://www.diracdelta.co.uk/science/source/s/p/spherical%20triangles/source.html (except for the area formula, which is not of any use here anyway).
Label the following points on the sphere: $A$ at London, $B$ at New York, $C$ at the north pole. Labeling the opposite sides of the spherical triangle $a$, $b$, $c$, assuming those are all angular measurements in radians, then $A$ (London) is $b$ radians from $C$ (the north pole) so the latitude of London is $lat_A = \frac\pi2 - b$; so $\sin(lat_A) = \cos b$ and $\cos(lat_A) = \sin b$. We can see then that the great circle formula is just one of the cosine formulas, which you correctly applied.
Once you have the distance $c$ from $A$ to $B$, it is relatively quick and easy to get the departure angles at New York and London by applying the sine law to $\triangle ABC$: $$ \frac{\sin A}{\sin a} = \frac{\sin B}{\sin b} = \frac{\sin C}{\sin c}. $$ Keep this in mind about the sine law: it's not one big equation, it's actually three equations, $\frac{\sin A}{\sin a} = \frac{\sin B}{\sin b}$, $\frac{\sin A}{\sin a} = \frac{\sin C}{\sin c}$, and $\frac{\sin B}{\sin b} = \frac{\sin C}{\sin c}$. We usually use only one of the three equations at a time.
You already know $a$, $b$, $C$, and $c$, so you can get $A$ from $\frac{\sin A}{\sin a} = \frac{\sin C}{\sin c}$ (one equation, one unknown) and $B$ from $\frac{\sin B}{\sin b} = \frac{\sin C}{\sin c}$.
Either angle $A$ or angle $B$ can help you find the maximum latitude of the great circle, but just one of them will be enough. Suppose we just figured out the angle at $A$. The trick then is to draw a new spherical triangle with vertices at $A$ and $C$, with the third vertex somewhere along the great-circle path from $A$ to $B$. Possible choices for the third vertex are the point where the great circle crosses the equator, or the point of maximum latitude itself.
Call the new vertex $B'$ and let $a'$ be the length of the arc from $B'$ to $C$. The side of $\triangle AB'C$ opposite $B'$ is still $b$ (the arc from $A$ to $C$ has not changed), and the side opposite $A$ is $a'$. The angle at $A$ also either has not changed, or it's $\frac\pi2$ minus the old angle; either way, its sine is still $\sin A$. So the sine law applied to $\triangle AB'C$ says that $$ \frac{\sin A}{\sin a'} = \frac{\sin B'}{\sin b}. $$
If you chose to put $B'$ at the maximum latitude, the angle of $\triangle AB'C$ at $B'$ must be a right angle; so you have three knowns ($\sin A$, $B'$, and $b$) and one unknown $a'$ to solve for. Then the latitude of $B'$ is $\frac\pi2 - a'$.
If you chose to put $B'$ at the crossing of the great circle and the equator, the arc $a'$ is $\frac\pi2$ (the arc from the equator to the pole); so you have three knowns ($\sin A$, $a'$, and $b$) and one unknown $B'$ to solve for. Then $\frac\pi2 - B'$ is the angle at which the great circle crosses the equator, which is also the maximum latitude on the great circle.