Find $c>0$ such that if $r$, $s$, and $t$ are the roots of the cubic $f(x)=x^3-4x^2+6x-c,$ then $$1=\dfrac1{r^2+s^2}+\dfrac1{s^2+t^2}+\dfrac1{t^2+r^2}.$$
I proceeded thus:$$r+s+t=4 ; rs+st+tr=6$$$$r^2+s^2+t^2=4$$
Also since $c>0$ so by Descarte's Rule we know that 2 roots are negative and one root is positive. But how to proceed further.
Also $r^2+s^2+t^2\ge rs+st+tr$ must be true since $r^2,s^2,t^2$ are positive but even that is not true. Please help.
On
There is probably a better way, but this one works. First, note that$$\frac1{r^2+s^2}+\frac1{s^2+t^2}+\frac1{t^2+r^2}=\frac{r^4+s^4+t^4+3 s^2 r^2+3 t^2 r^2+3 s^2 t^2}{\left(r^2+s^2\right) \left(r^2+t^2\right) \left(s^2+t^2\right)}.\tag1$$Now, you can express both the numerator and the denominator of this expression as polynomials in $r+s+t$, $rs+rt+st$ and $rst$: the numerator is equal to$$(r+s+t)^4-4 (r s+t s+r t) (r+s+t)^2-2 r s t (r+s+t)+5 (r s+t s+r t)^2$$and the denominator is equal to\begin{multline}-2 r s t (r+s+t)^3+(r s+t s+r t)^2 (r+s+t)^2+\\+4 r s t (r s+t s+r t) (r+s+t)-2 (r s+t s+r t)^3-(rst)^2\end{multline}Now, replace $r+s+t$, $rs+rt+st$ and $rst$ with $4$, $6$, and $c$ respectively. If you express the fact that the expression $(1)$ is equal to $1$, you will get then a quadratic equation in $c$.
On
Since we know that $r^2 + s^2 + t^2= 4$, then $ 1 = \dfrac1{4-r^2} + \dfrac1{4-s^2} + \dfrac1{4-t^2} $.
If $f(x)=0$ has roots $r,s,t$, then $f\left(\sqrt x\right)=0 $ has roots $r^2,s^2,t^2$.
$$\begin{eqnarray} f\left( \sqrt x \right) &=& 0 \\ x \sqrt x - 4x + 6\sqrt x - c &=&0 \\ \sqrt x (x + 6) &=& 4x + c \\ x (x+6)^2 &=& (4x+c)^2 \\ x^3 - 4x^2 + x (36-8c) - c^2 &=& 0 \end{eqnarray} $$
Thus, $g(x) := x^3 - 4x^2 + x (36-8c) - c^2 $ has roots $r^2,s^2,t^2$.
And $h(x) := g(-x) = -x^3 - 4x^2 + x(8c-36) - c^2 $ has roots $-r^2,-s^2,-t^2$.
Similarly,
$$ j(x) := h(x-4) = -(x-4)^3 - 4(x-4)^2 + (x-4)(8c-36) - c^2 \\ = -x^3 + 8x^2 + x(8c-52) + (144-c^2-32c) $$
has roots $4-r^2,4-s^2,4-t^2$.
Hence, by Vieta's, $ 1 = \dfrac{52-8c}{144-c^2-32c} \Rightarrow c = -12+ 2 \sqrt{59} $ (take the positive root only).
I do believe the best answer is by combining both ideas.
$$ 1=\frac{1}{4-r^2}+\frac{1}{4-s^2}+\frac{1}{4-t^2} $$
Putting all the fractions together gives
$$ 1 = \frac{8 (r^2+s^2+t^2) - (rs)^2 - (st)^2 - (tr)^2-48}{16 (r^2+s^2+t^2) - 4 (rs)^2 -4 (st)^2 -4 (tr)^2 -64 + (rst)^2} $$
We know from $f(x)=(x-r)(x-s)(x-t)$ that
As the OP observed before, this leads to
$r^2+s^2+t^2=(r + s + t)^2 - 2 (r s + s t + t r)=4$
$(rs)^2 + (st)^2 + (tr)^2 = (rs + st + tr)^2-2 r s t (r + s + t)=36-8c$
With these two new rules you find directly that
$$1=\frac{8c-52}{ c^2+32c-144}$$
Which you can solve to $c=2\sqrt{59}-12$.