Given are $x$, $y$ and $z\ge0$, if $x^2+y^2+z^2+2xyz=1$, show that \[x+y+z\le\dfrac32.\] I want to solve this by quadratic function.
Let $x+y+z=k$, so $z=k-x-y$. Plug in and expand to have \[-2 x {{y}^{2}}+2 {{y}^{2}}-2 {{x}^{2}} y+2 k x y+2 x y-2 k y+2 {{x}^{2}}-2 k x+{{k}^{2}}-1=0.\] See this as a quadratic function of $y$, so $\Delta\ge0$, which implies that \begin{align}4 {{x}^{4}}-8 k {{x}^{3}}+8 {{x}^{3}}+4 {{k}^{2}} {{x}^{2}}-12 {{x}^{2}}+8 k x-8 x-4 {{k}^{2}}+8\ge0.\tag{1}\end{align} We need to find the minimum of $k$, so that $\exists~x$ for $(1)$ to be true. When $k=\dfrac32$, \[4 {{x}^{4}}-4 {{x}^{3}}-3 {{x}^{2}}+4 x-1=0\implies\left[ x=1\operatorname{,}x=-1\operatorname{,}x=\frac{1}{2}\right]\] are all solutions of $\Delta=0$. How to prove $k\ge\dfrac32$ don't work?
If $x = 1$, clearly the desired inequality is true.
In the following, assume that $x < 1$.
Proceeding along OP's approach, we have $$\Delta = 4(1-x^2)[3 - 2k -(x-k+1)^2] \ge 0$$ which results in $k \le 3/2.$