I have a problem: Find the maximum value of $P=\frac{x^3y^4z^3}{(x^4+y^4)(xy+z^2)^3}+\frac{y^3z^4x^3}{(y^4+z^4)(yz+x^2)^3}+\frac{z^3x^4y^3}{(z^4+x^4)(zx+y^2)^3}$ with $x,y,z>0$. Is there anyway to solve this problem by using the AM-GM inequality ? Thank for your answer.
The first way, I tried to use AM-GM twice at the two sum in the denominator, but I get the sum of the fractions with square roots of $xy, yz, zx$ at its denominators.
The other way, I tried to factor the denominator and use both AM-GM and the Schwarz inequality but still get the same with the first way, with higher order
For $x=y=z=1$ we get $P=\frac{3}{16}.$
We'll prove that it's a maximal value.
Indeed, by AM-GM $$\sum_{cyc}\frac{x^3y^4z^3}{(x^4+y^4)(xy+z^2)^3}\leq\sum_{cyc}\frac{x^3y^4z^3}{(x^4+y^4)(2\sqrt{xyz^2})^3}=\sum_{cyc}\frac{\sqrt{x^3y^5}}{8(x^4+y^4)}.$$ Let $\sqrt{\frac{x}{y}}=a$, $\sqrt{\frac{y}{z}}=b$ and $\sqrt{\frac{z}{x}}=c$.
Thus, $abc=1$ and we need to prove that $$\sum_{cyc}\frac{a^3}{a^8+1}\leq\frac{3}{2}$$ or $$\sum_{cyc}\left(\frac{1}{2}-\frac{a^3}{a^8+1}\right)\geq0$$ or $$\sum_{cyc}\left(\frac{1}{2}-\frac{a^3}{a^8+1}-\frac{1}{2}\ln{a}\right)\geq0.$$ Now, let $f(a)=\frac{1}{2}-\frac{a^3}{a^8+1}-\frac{1}{2}\ln{a}.$
Thus, $$f'(a)=-\tfrac{(a-1)(a^{11}(a^4+a^3+a^2+a+1)-9a^8(a^2+a+1)-7a^3(a^4+a^3+a^2+a+1)-a^2-a-1)}{2a(a^8+1)^2}.$$ Since by the Descartes' rule of signs the polynomial $$ a^{11}(a^4+a^3+a^2+a+1)-9a^8(a^2+a+1)-7a^3(a^4+a^3+a^2+a+1)-a^2-a-1$$ has an unique positive root (this root is $a_1=1.56...$) and $f(a_1)>0$,
we see that in $a_1$ the function $f$ has a local maximum,
which gives that$f$ decreases on $[a_1,+\infty)$ and there is an unique $a_0>a_1$, for which $f(a_0)=0$.
Easy to see that $a_0=2.679...$ and since $a_{min}=1$ and $f(1)=0$,
our inequality is proven for $\max\{a,b,c\}\leq2.5$
Let $a\geq2.5$.
Also, by AM-GM $$\frac{x^3}{x^8+1}=\frac{x^3}{3\cdot\frac{x^8}{3}+5\cdot\frac{1}{5}}\leq\frac{x^3}{8\sqrt[8]{\left(\frac{x^8}{3}\right)^3\left(\frac{1}{5}\right)^5}}=\frac{1}{8}\sqrt[8]{3^35^5}.$$ Id est, $$\sum_{cyc}\frac{a^3}{a^8+1}\leq\frac{1}{4}\sqrt[8]{3^35^5}+\frac{2.5^3}{2.5^8+1}=1.0423...<\frac{3}{2}$$ and we are done!