Find the maximum of $P$

Question

Given $a, b, c \geq 0$ and $a + b + c = 6$. Find the maximum of:

$$P = (a-b)(b-c)(c-a)$$

My guess is the maximum of $P$ would be just $0$ but I don't know whether this guess is true

And if it's true, how do I prove it?

Answer 1

Hint: Since the expression is cyclic, WLOG Either $ a \geq b \geq c $ or $ a \leq b \leq c$.
In the former, $P \leq 0$, so we may assume the latter case.

$$ P = |b-a| \times |c - b | \times | c - a |$$

Hint: Replacing $ (a, b, c) \rightarrow ( 0, b, c+a )$ increases the first and third terms while keeping the second term constant, hence increases $P$. So, we may assume $ a = 0, b+c = 6$.

Hence, we just need to maximize $ (6-c) ( 2c-6 ) c$ subject to $ 3 \leq c \leq 6$, which is a cubic.
It is left to the reader to show that the max occurs when $ c = 3 + \sqrt{3} $ using their favorite approach.

Answer 2

Suppose $c = \min\{a,b,c\},$ so $$(a-c)^2 \leqslant a^2, \quad (b-c)^2 \leqslant b^2.$$ By the AM-GM inequality we have $$\begin{aligned}P^2 = (a-b)^2(b-c)^2(c-a)^2 & \leqslant a^2b^2(a-b)^2 = \frac14 \cdot 2ab \cdot 2ab \cdot (a-b)^2 \\& \leqslant \frac14\left(\frac{2ab+2ab+(a-b)^2}{3}\right) ^3\\&=\frac{(a+b)^6}{108}\leqslant \frac{(a+b+c)^6}{108}\end{aligned}$$ Therefore $$(a+b+c)^3 \geqslant 6 \sqrt 3 (a-b)(b-c)(c-a).$$ Equality occur when $a^2+b^2=4ab,\,c= 0$ and permutation.

Because $a+b+c=6,$ so $$(a-b)(b-c)(c-a) \leqslant \frac{(a+b+c)^3}{6\sqrt 3} = 12\sqrt 3.$$ Equality occur when $$a^2+b^2=4ab,\,c= 0,\,a+b+c=6. \quad (1)$$ Solve $(1)$ we get $a = 3+\sqrt3,\,b = 3-\sqrt3,\, c = 0$ or $a = 3-\sqrt3, \,b = 3+\sqrt3,\, c = 0.$

Thefore $P_{\max} = 12\sqrt 3.$