Let $a,b,c\in R^+$ prove that the inequality $$\sqrt{\frac{a^2}{6a^2+5ab+b^2}}+\sqrt{\frac{b^2}{6b^2+5bc+c^2}}+\sqrt{\frac{c^2}{6c^2+5ca+a^2}}\le \frac{\sqrt{3}}{2}$$
My try:$$\sum\limits_{cyc} \sqrt{\frac{a^2}{6a^2+5ab+b^2}}=\sum\limits_{cyc} \sqrt{\frac{a^2}{\left(3a+b\right)\left(2a+b\right)}}=\sum\limits_{cyc} \sqrt{\frac{1}{\left(3+\frac{b}{a}\right)\left(2+\frac{b}{a}\right)}}=\sum\limits_{cyc} \frac{1}{\sqrt{\left(x+3\right)\left(x+2\right)}}\le \sum\limits_{cyc} \frac{1}{4\sqrt{3}}\left(\frac{4}{x+3}+\frac{3}{x+2}\right),$$ where $\frac{b}{a}=x;\frac{c}{b}=y;\frac{a}{c}=z\left(xyz=1\right)$ It is easy now but i also solve it by SOS but stuck, here is my solution by SOS and Cauchy-Schwarz
$$\left(\sum_{cyc}\sqrt{\frac{a^2}{6a^2+5ab+b^2}}\right)^2\le 3\left(\sum_{cyc} \frac{a^2}{6a^2+5ab+b^2}\right)\le \frac{3}{4}$$
Or $$\sum_{cyc} \left(\frac{a^2}{6a^2+5ab+b^2}-\frac{1}{12}\right)\le 0\Leftrightarrow \sum_{cyc} \frac{\left(a-b\right)\left(6a+b\right)}{12\left(2a+b\right)\left(3a+b\right)}\le 0$$
And i tried to taking $6a^2-5ab-b^2$ into $(a-b)\cdot Q(a,b,c)-(c-a)\cdot P(a,b,c)$ but unsuccessful
I want to solve it by SOS, help.
SOS works for symmetric inequalities. For cyclic inequalities SOS helps sometimes,
but for your inequality it does not help or at least, does not help immediately.
Also, your second way gives a wrong inequality.
Try $c=0.0001$, $b=0.01$ and $a=1$.
Your first idea gives a proof.
Indeed, by AM-GM $$\sum_{cyc}\frac{a}{\sqrt{6a^2+5ab+b^2}}=\sum_{cyc}\left(\frac{a}{2\sqrt3}\cdot\sqrt{\frac{1}{\frac{3a+b}{4}}\cdot\frac{1}{\frac{2a+b}{3}}}\right)\leq\sum_{cyc}\frac{a}{4\sqrt3}\left(\frac{4}{3a+b}+\frac{3}{2a+b}\right).$$ Thus, it's enough to prove that $$\sum_{cyc}\left(\frac{4a}{3a+b}+\frac{3a}{2a+b}\right)\leq6$$ or $$\sum_{cyc}(12a^4b^2+19a^4c^2+53a^3b^3+53a^4bc-112a^3b^2c+78a^3c^2b-103a^2b^2c^2)\geq0,$$ which is true by SOS, Rearrangement and AM-GM.
For example, $$\sum_{cyc}(a^4c^2-a^3b^2c)=a^2b^2c^2\sum_{cyc}\left(\frac{a^2}{b^2}-\frac{a}{c}\right)=$$ $$=\frac{1}{2}a^2b^2c^2\sum_{cyc}\left(\frac{a^2}{b^2}+\frac{b^2}{c^2}-\frac{2a}{c}\right)=\frac{1}{2}a^2b^2c^2\sum_{cyc}\left(\frac{a}{b}-\frac{b}{c}\right)^2\geq0.$$ Can you end it now?
Also, the Tangent Line method helps.
Let $\frac{b}{a}=x$, $\frac{c}{b}=y$ and $\frac{a}{c}=z$.
Thus, $xyz=1$ and $$\frac{\sqrt3}{2}-\sum_{cyc}\sqrt{\frac{a^2}{6a^2+5ab+b^2}}=\sum_{cyc}\left(\frac{1}{2\sqrt3}-\frac{1}{\sqrt{x^2+5x+6}}\right)=$$ $$=\sum_{cyc}\left(\frac{1}{2\sqrt3}-\frac{1}{\sqrt{x^2+5x+6}}-\frac{7}{48\sqrt3}\ln{x}\right)\geq0$$ because easy to show that $$\frac{1}{2\sqrt3}-\frac{1}{\sqrt{x^2+5x+6}}-\frac{7}{48\sqrt3}\ln{x}\geq0$$ for all $x>0$.