Given a positive semi-definite (PSD) matrix $M$ =: $$\begin{bmatrix} 1 & a & c \\ a & 1 & b \\ c & b & 1 \end{bmatrix}$$, how to come up with the constraint: $$ab - \sqrt{(1-a^2)(1-b^2)} \le c \le ab + \sqrt{(1-a^2)(1-b^2)}$$?
Update:
It turns out to get the above constraint, we only need to use a criterion (https://en.wikipedia.org/wiki/Sylvester%27s_criterion) -- a PSD matrix must have a non-negative determinant. Thank to all the great answers below!
On
Hint and observation: For positive definiteness we must have $1-a^2>0$, hence there exists a number $\phi\in[0,pi)$ such that $a=cos(\phi)$. Furthermore we want to have $$\det(M)=1-a^2-b^2-c^2+2abc>0,$$ that is $$1-\cos^2(\phi)>b^2+c^2-2bc\cos(\phi).$$ Consider the triangle with edges $b$ and $c$ with inscribed angle $\phi$. Call its third edge $u$. Then we know $$\sin^2(\phi)=u^2,$$ that is $\sin(\phi)=u$ for some non-negative number $u$. For $\sin(\phi)= u$ the law of Sine tells us that $u/\sin(\phi)=1$, hence the vertices of the triangle lie inside a circle with diameter $1$.
HINT:
The matrix is positive semi-definite, if and only if it is the Gram matrix of three vectors of norm $1$. Now write $a$, $b$, $c$ as $\cos u$, $\cos v$, $\cos w$.
Side note: the determinant of the matrix equals $$\Delta = 4 \sin s \sin(s-u) \sin (s-v)\sin (s-w)$$ where $2 s = u+v+w$.
${\bf Added:}$ Using only algebra: the conditions are $|a|,|b|,|c|\le 1$ and the determinant $$1-(a^2 + b^2 + c^2 ) + 2 a b c \ge 0$$
The equation in $c$ $$1-(a^2 + b^2 + c^2 ) + 2 a b c = 0$$ has solutions $$c_{1,2} = a b \pm \sqrt{(1-a^2)(1-b^2)}$$