$S$ is the unit sphere without its north and south poles. $H$ is the hyperboloid $x^2+y^2-z^2=1$. For $p\in S$, $l_p$ is the ray perpendicular to the $z$ axis that starts on the $z$ axis and passes through $p$. $F(p)=l_p\cap H$. How can I prove that $F$ is a differentiable function?
So I let $p=(a,b,c)$ so the ray is $\{(x,y,z)|y=\frac{b}{a}x,z=c\}$. But there would be two intersections with $H$. How canI avoid that and proceed from here?
In such a case, the usual parameterization is by cylindrical coordinates $(R,\theta,z)$ with $\theta$ the polar angle (aka longitude), $0 \leq \theta < 2 \pi$); in this way, there are no "sign ambiguities".
For an "altitude" $c$ with $-1 < c < 1$, the current point on the (unit) sphere is:
$$(a,b,c)=(\sqrt{1-c^2}\cos\theta,\sqrt{1-c^2}\sin\theta,c) \ \ (1)$$
(check that $a^2+b^2+c^2=1$).
Your "radial projection" amounts to find a multiplier $m$ such that $(ma,mb,c)$ belongs to the hyperboloid.
Imposing: $(ma)^2+(mb)^2-c^2=1$ and taking into account (1), it finally gives: $$m=\sqrt{\dfrac{1+c^2}{1-c^2}} \ \ (2)$$
Thus, $F$ can be expressed in the following way with $m$ given by (2):
$$F:(a,b,c) \rightarrow (ma,mb,c)$$
where all the involved expressions are differentiable (even $C^{\infty}$).