For the surface
$$\{ (x_1, x_2, x_3) \in \mathbb R^3 \mid -x_1^2+x_2^2-x_3^2 + 10x_1x_3=1 \}$$
I want to find the points on it closest to the origin. I know how to do a substantial part of this. First I make the matrix
$$A = \begin{pmatrix}-1 & 0 & 5 \\ 0 & 1 & 0 \\ 5 & 0 & -1\end{pmatrix}$$
and then get the eigenvalues
$$|A-\lambda I| = \begin{vmatrix} -1-\lambda & 0 & 5 \\ 0 & 1-\lambda & 0 \\ 5 & 0 & -1-\lambda\end{vmatrix}$$
$$ = (-1-\lambda)(1-\lambda)(-1-\lambda) + 5(-5(1-\lambda))$$
$$= (1-\lambda)([-1-\lambda]^2-25)$$
$$= (1-\lambda)(-1-\lambda-5)(-1-\lambda+5)$$
$$= (1-\lambda)(-6-\lambda)(-4-\lambda) $$
$$= (1-\lambda)(6+\lambda)(4+\lambda)$$
So the eigenvalues are $\lambda = 1,-6,-4$ and therefore the quadratic form can be represented as
$$c_1^2 - 6c_2^2 - 4c_3^2=1$$
The part I don't get is how to find the point closest to the origin from here. I've seen solutions to problems similar to this where they merely set two variables to 0 and out of all such choices of two variables, they pick the one with the least value for the third which satisfies the equation.
However, how do we know this process will select the point closest to the origin? If coefficients were made suitably extreme, at least on the face of it, you'd think we might be able to engineer and example where picking some very small mixture of non-zero values of $c_1, c_2, c_3$ would satisfy the equation and be smaller than what you'd get for setting some two of them to 0.
In this particular case we could set $c_2=c_3=0$ so that $c_1=1$. But what guarantee is there that we couldn't give $c_2$ just a little something, drastically reducing the amount we need to give to $c_1$? Even if we couldn't do that here, what if the coefficient of $c_2$ were negative a million, so that you'd think just a little bit in $c_2$ goes a very long way to reducing what we need from $c_1$?
[Edit: As pointed out, I should have said that the characteristic equation was
$$(1-\lambda)(-6-\lambda)(4-\lambda)$$
so that the eigenvalues are 1, -6, and 4. Then the quadratic can be expressed as
$$c_1^2-6c_2^2+4c_3^2=1$$]
If you start at $(c_1,c_2,c_3)=(1,0,0)$ and give a little something to $c_2$, then you would have to increase $c_1$ in order to keep $c_1^2 - 4 c_2^2 - 6 c_3^2$ equal to $1$. (I'm not quite sure I understand your worries correctly, but you seem to think that $c_1$ can be decreased? If so, you've got it backwards.)
And any point $(c_1,c_2,c_3)$ with $c_1>1$ is farther from the origin than $(c_1,c_2,c_3)=(1,0,0)$, so $(1,0,0)$ is closest (together with $(-1,0,0)$).