Question. Let $a,b,c>0: abc=a+b+c+2.$ Prove that$$2\left(\sqrt{ab-1}+\sqrt{bc-1}+\sqrt{ca-1}\right)\le \left(\sqrt{a}+\sqrt{b}+\sqrt{c}\right)\sqrt{\sqrt{ab}+\sqrt{bc}+\sqrt{ca}}.$$I am looking for a nice proof by hand, for which there is a possibility to find this proof during a competition.
Equality holds at $a=b=c=2.$
I've tried to use the well-known substitution without success.
Indeed, by the given condition we easily get $$\frac{1}{a+1}+\frac{1}{b+1}+\frac{1}{c+1}=1.$$ Now, if we set $a=\dfrac{y+z}{x};b=\dfrac{x+z}{y}$ then $c=\dfrac{y+x}{z}.$ Replace the set to OP, it's very complicated.
Hope to see some ideas. All contributions are welcome.
I hope the follwing will help.
By squaring both side, we'll prove that $$4(ab+bc+ca-3)+8\sum_{cyc}\sqrt{(ab-1)(bc-1)}\le \left(\sqrt{a}+\sqrt{b}+\sqrt{c}\right)^2\left(\sqrt{ab}+\sqrt{bc}+\sqrt{ca}\right) \tag{*}.$$ Now, by Schur of third degree inequality $$\sqrt{a}+\sqrt{b}+\sqrt{c}+\frac{9\sqrt{abc}}{\sqrt{a}+\sqrt{b}+\sqrt{c}}\ge 2\left(\sqrt{ab}+\sqrt{bc}+\sqrt{ca}\right),$$or $$\left(\sqrt{a}+\sqrt{b}+\sqrt{c}\right)^2\ge 4\left(\sqrt{ab}+\sqrt{bc}+\sqrt{ca}\right)-\frac{9\sqrt{abc}}{\sqrt{a}+\sqrt{b}+\sqrt{c}}>0.$$ Thus, we prove stronger inequality of $(*)$ $$4(ab+bc+ca-3)+8\sum_{cyc}\sqrt{(ab-1)(bc-1)}\le 4\left(\sqrt{ab}+\sqrt{bc}+\sqrt{ca}\right)^2-\frac{9\sqrt{abc}\left(\sqrt{ab}+\sqrt{bc}+\sqrt{ca}\right)}{\sqrt{a}+\sqrt{b}+\sqrt{c}},$$ or $$\frac{9\sqrt{abc}\left(\sqrt{ab}+\sqrt{bc}+\sqrt{ca}\right)}{\sqrt{a}+\sqrt{b}+\sqrt{c}}+8\sum_{cyc}\sqrt{(ab-1)(bc-1)}\le 8\sqrt{abc}\left(\sqrt{a}+\sqrt{b}+\sqrt{c}\right)+12 \tag{**}.$$
Now, use your subtitution $a=\dfrac{y+x}{z};b=\dfrac{y+z}{x}$ and $c=\dfrac{z+x}{y}$ where $x,y,z>0.$ It implies $ab-1=\dfrac{y(x+y+z)}{xz};bc-1=\dfrac{z(x+y+z)}{xy};ca-1=\dfrac{x(x+y+z)}{yz}.$
Hence, by AM-GM $$8\sum_{cyc}\sqrt{(ab-1)(bc-1)}=8\sum_{cyc}\frac{(x+y+z)}{\sqrt{xy}}\le 8\sum_{cyc}\frac{(x+y)}{\sqrt{xy}}+4\sum_{cyc}\frac{x+y}{z}.\tag{1}$$ Also, by C-S \begin{align*} 8\sqrt{abc}\left(\sqrt{a}+\sqrt{b}+\sqrt{c}\right)&=8\sum_{cyc}\frac{(y+x)\sqrt{(z+y)(x+z)}}{z\sqrt{yx}}\\&\ge 8\sum_{cyc}\frac{(y+x)(z+\sqrt{yx})}{z\sqrt{yx}}\\&=8\sum_{cyc}\frac{x+y}{\sqrt{xy}}+8\sum_{cyc}\frac{x+y}{z}. \tag{2} \end{align*} From $(1)$ and $(2)$ it remains to prove $$4\sum_{cyc}\frac{x+y}{z}+12\ge \frac{9\sqrt{abc}\left(\sqrt{ab}+\sqrt{bc}+\sqrt{ca}\right)}{\sqrt{a}+\sqrt{b}+\sqrt{c}}.$$ Notice that by C-S \begin{align*} 4\sum_{cyc}\frac{x+y}{z}+12&=2\left[(x+y)+(y+z)+(z+x)\right]\cdot\left(\frac{1}{x}+\frac{1}{y}+\frac{1}{z}\right) \\& \ge 2\left(\sum_{cyc}\sqrt{\frac{x+y}{z}}\right)^2\\&=2\left(\sqrt{a}+\sqrt{b}+\sqrt{c}\right)^2. \end{align*} Thus, we need to prove $$2\left(\sqrt{a}+\sqrt{b}+\sqrt{c}\right)^3\ge 9\sqrt{abc}\left(\sqrt{ab}+\sqrt{bc}+\sqrt{ca}\right).$$ Since $a+b+c=2\sum_{cyc}\dfrac{x+y}{z}\ge 6$ then $abc=a+b+c+2\le \dfrac{4(a+b+c)}{3}.$
Finally, we prove $$\left(\sqrt{a}+\sqrt{b}+\sqrt{c}\right)^3\ge 3\sqrt{3(a+b+c)}\left(\sqrt{ab}+\sqrt{bc}+\sqrt{ca}\right),$$which is true by AM-GM \begin{align*} \left(\sqrt{a}+\sqrt{b}+\sqrt{c}\right)^3&=\sqrt{\left[a+b+c+2\left(\sqrt{ab}+\sqrt{bc}+\sqrt{ca}\right)\right]^3}\\&\ge \sqrt{27(a+b+c)\left(\sqrt{ab}+\sqrt{bc}+\sqrt{ca}\right)^2}\\&= 3\sqrt{3(a+b+c)}\left(\sqrt{ab}+\sqrt{bc}+\sqrt{ca}\right). \end{align*} Hence, the proof is done.