Solving an applied problem I was faced with the need to find supremum of the following function $$f(x,y,z)=\frac{(x-xyz)(y-xyz)(z-xyz)}{(1-xyz)^3}$$ where $f\colon\ [0,1]^3\backslash\{(1,1,1)\} \to\mathbb{R}$.
I used Wolfram Mathematica and it resulted with "there is no maximum in the region in which the objective function is defined" and that the supremum is $\frac{8}{27}$ when $x,y,z\to 1$. I was able to verify some of it: $$\lim\limits_{x,y,z\to 1}f(x,y,z) = \lim\limits_{x\to 1}f(x,x,x) = \lim\limits_{x\to 1}\frac{(x-x^3)^3}{(1-x^3)^3} = \lim\limits_{x\to 1}\frac{x^3(x+1)^3}{(x^2+x+1)^3} = \frac{8}{27} $$ though I'm not sure how to justify the first equality. Can I neglect direction of derivative in this case?
And secondly, I don't know how to show that $f(x,y,z)< \frac{8}{27}$ for all $x,y,z\in[0,1]^3\backslash\{(1,1,1)\}$. When I tried to do that straightforwardly I drowned in computations.
You want to find the maximum inside the unit cube of $$f(x, y, z) = \frac{xyz(1-xy)(1-yz)(1-zx)}{(1-xyz)^3}$$
Now, suppose this maximum is when WLOG $y > z$. Then we have $$f(x, y, z) = \frac{xyz(1-yz)(1-x(y+z)+x^2yz)}{(1-xyz)^3} < f(x, \sqrt{yz}, \sqrt{yz} )$$ as $y+z > 2\sqrt{yz}$. Thus we must have $x=y=z$ for the maximum. However, $$f(t, t, t) = \frac{t^3(1-t^2)^3}{(1-t^3)^3}=\frac{t^3(1+t)^3}{(1+t+t^2)^3}< \frac8{27}$$ as $27t^3(1+t)^3 < 8(1+t+t^2)^3 \iff 3t(1+t)< 2(1+t+t^2) \iff (1-t)(t+2) > 0$.
Finally we note that as $t \to 1$, $f(t, t, t)$ gets arbitrarily close to $ \frac8{27}$ so this is the supremum.