I have the equation $F=u_x^2 + u_y^2 - 1$ and that $u=0$ on the boundary of the ellipse $\frac{x^2}{a^2} +\frac{y^2}{b^2}=1$
At $u=0$ the ellipse can be parametrised as $x=a\cos(s), y=b\sin(s)$
Charpits eqs give me:
$dx/dt= 2p$
$dy/dt = 2q$
$dp/dt = dq/dt = 0$
$du/dt = 2p^2 + 2q^2$
where $p = \frac{\partial u}{\partial x}$ and $q = \frac{\partial u}{\partial y}$
solving charpits eqs I get:
$p = p_0(s)$
$q = q_0(s)$
$x = 2p_0(s)t + a\cos(s)$
$y = 2q_0(s)t + b\sin(s)$
Since $F(p,q) = p^2 + q^2 - 1$, this implies that $p_0^2 + q_0^2 = 1$
To find the envelope i need to find when the jacobian =0 which is when $2bp_0\cos(s)+ 2aq_0\sin(s) = 0$
I am confused as to how to draw the envelope of rays that propagate into the ellipse and also how to determine the ridge of discontinuity
I think you might have missed a sign in $\dot x_0(s)=-a\sin(s)$. Then the equation for $p_0,q_0$ reads as $$ 0=\dot z_0(s)=p_0(s)\dot x_0(s)+q_0(s)\dot y_0(s)=-ap_0\sin(s)+bq_0\cos(s). $$ This tells us that the unit vector $(p_0,q_0)$ has to be orthogonal to $(-a\sin(s), b\cos(s))$, thus $$(p_0,q_0)=\pm\frac{(b\cos(s),a\sin(s))}{\sqrt{b^2\cos^2(s)+a^2\sin^2(s)}}.$$ As this should point inwards, chose the negative sign.
To find the characteristic that some point $(x,y)$ lies on, one has to solve $$ \left.\begin{aligned} x&=x_0+2p_0t=a\cos(s) - r b \cos(s)\\ y&=y_0+2q_0t=b\sin(s) - r a \sin(s) \end{aligned}\right\} \implies \frac{x^2}{(a-rb)^2}+\frac{y^2}{(b-ar)^2}=1 $$ where $r$ is some positive multiple of $t$. The last is equivalent to a 4th degree polynomial equation, by the intermediate value theorem there is at least one solution between $r=0$ and $\min(\frac ba,\frac ab)$, there is another solution for $r>\max(\frac ba,\frac ab)$, but that might be inadmissible.
Two close-by rays associated to the angles $s$ and $s+ds$ intersect at $$ \left.\begin{aligned} (a - r b) \cos(s) = (a - (r+dr) b) \cos(s+ds)\\ (b - r a) \sin(s) = (b - (r+dr) a) \sin(s+ds) \end{aligned}\right\} \implies \left\{\begin{aligned} 0=-b\,dr\,\cos(s)-(a-br)\sin(s)ds\\ 0=-a\,dr\,\sin(s)+(b-ar)\cos(s)ds \end{aligned}\right. $$ so that necessarily $$a(a-br)\sin^2(s)+b(b-ar)\cos^2(s)=0\implies r = \frac{a^2\sin^2s+b^2\cos^2s}{ab}$$
Plotting that curve against the ellipse gives the picture