I know that to find the point on $6x^2+y^2=262090$ that is nearest to the point $(1045,0)$, we can try to minimize the squared distance $S=(x-1045)^2+262090-6x^2$. However, calculus tells us that this function does not have a minimum point (instead only a maximum point exists).
But if we try to minimize the distance function (without squaring) then we can find the minimum.
So, When exactly can we actually square the distance function to find the max/min point for distance problems?
There is nothing wrong with using the squared distance.
You converted the problem into a one-parameter minimization problem. You are looking for the minimum of a smooth function on the interval $[-\sqrt{262090/6},\sqrt{262090/6}]$. The minimum value is obtained at a zero of the derivative (a critical point) or at one of the endpoints. The function is a downward opening parabola, so you know that any critical point is a local maximum. Therefore you have to look at the endpoints.
Alternatively, you could have converted the problem into minimization over $y$. You can solve that $x=\pm\sqrt{(262090-y^2)/6}$. If you draw a picture, it becomes clear that the closest point must be in the right half of the ellipse. (If you don't believe in pictures, you can treat the two halves separately.) In this half $x>0$.
This leads to the squared distance being $$ \begin{split} &(x-1045)^2+(y-0)^2 \\=& \frac16(262090-y^2)-2090\sqrt{(262090-y^2)/6}+1045^2+y^2 \\=& \frac56y^2-2090\sqrt{(262090-y^2)/6}+1045+262090/6. \end{split} $$ Now each term is at its smallest when $y=0$, so the minimum is at $y=0$. The corresponding value of $x$ is $\sqrt{262090/6}$ — the endpoint of the $x$-interval!