$(0,0)$ with semi-major axis a and semi-minor axis $b$">
I want find out for each point $(x,y)$ in the white region, the nearest point $(a \cos \theta, b \sin \theta )$ on the ellipse curve.I tried with the following approach. $$ \text {distance} = \sqrt{(x-a \cos \theta)^2+(y-b \sin \theta)^2} $$ I find out the first derivative of the $$F(\theta)=(x-a \cos \theta)^2+(y-b \sin \theta)^2.$$ The equation after first derivative is: $$ ax \sec \theta- by \csc \theta=a^2-b^2. $$ How to calculate theta for this equation?
You can get it by solving a constrained optimization problem. You want the point $(u,v)$ that minimizes $f(u,v)=(u-x)^2+(v-y)^2$ subject to the constraint $\frac{u^2}{a^2} + \frac{v^2}{b^2} = 1$. The point $(x,y)$ doesn't need to be inside the ellipse.
The Lagrangian is given by $$ L(u,v,\lambda) = (u-x)^2+(v-y)^2-\lambda\left(\frac{u^2}{a^2} + \frac{v^2}{b^2}-1\right) $$
Now you just need to compute the critical points of $L$ and choose the ones that yield the smallest distance to $(x,y)$.
You can solve this by hand but, out of laziness, I give you Wolfram's solution... Two critical points
WARNING: Wolfram is not giving the full set of solutions. After computing $u,v$ in terms of $\lambda$ from the first two equations of the system $\nabla L = 0$ and substituting them in the last equation we get a fourth degree polynomial equation in $\lambda$. The solution must be conmputed in an alternative way if this polynomial has four real roots.
$$ \left( -\frac{a \sqrt{b^2-y^2}}{b}, \frac{-a^2 b^4 y+a^2 b^2 y^3+a b^3 x y \sqrt{b^2-y^2}+b^6 y-b^4 y^3}{a^4 b^2-a^4 y^2-2 a^2 b^4-a^2 b^2 x^2+2 a^2 b^2 y^2+b^6-b^4 y^2}\right) $$
and
$$ \left( \frac{a \sqrt{b^2-y^2}}{b}, \frac{-a^2 b^4 y+a^2 b^2 y^3-a b^3 x y \sqrt{b^2-y^2}+b^6 y-b^4 y^3}{a^4 b^2-a^4 y^2-2 a^2 b^4-a^2 b^2 x^2+2 a^2 b^2 y^2+b^6-b^4 y^2}\right) $$
One will correspond to the minimum distance and the other will correspond to the maximum distance.
This is not relevant to your question, but the Lagrange multipliers are $$ \frac{a^2 b^2-a^2 y^2+a b x \sqrt{b^2-y^2}}{b^2-y^2} $$
and
$$ \frac{a^2 b^2-a^2 y^2-a b x \sqrt{b^2-y^2}}{b^2-y^2}, $$
respectively.