Here's what I have tried:
I multiplied the second equation by $\sqrt[3]{abc}$ and get
$$\sqrt[3]{a^5c}+\sqrt[3]{b^5a}+\sqrt[3]{c^5b}=0$$
By manipulating the first expression, I obtain
$$ {1\over a^4b^4c^4}(a^5c+b^5a+c^5b)^2 $$
For simplicity, I attempt to let $x^3=a^5c,y^3=b^5a,z^3=c^5b$, and, as a result, the problem becomes the following:
Maximize $\displaystyle{\left(x^3+y^3+z^3\over xyz\right)^2}$ subjected to $x+y+z=0$
I know that this problem is likely to be solved by AM-GM inequality, so I wonder if anybody could provide me some help.
Because $x+y+z=0$, $z=-x-y$, and the objective function $f(x,y,z)$ becomes
$$ f(x,y,z)\triangleq\left(x^3+y^3+z^3\over xyz\right)^2=\left[x^3+y^3-(x+y)^3\over xy(-x-y)\right]^2 $$
According to binomial theorem, we have $(x+y)^3=x^3+3x^2y+3xy^2+y^3$, so we can further simplify $f(x,y,z)$ into
$$ \begin{aligned} f(x,y,z) &=\left[-3x^2y-3xy^2\over xy(-x-y)\right]^2 \\ &=\left[3x^2y+3xy^2\over xy(x+y)\right]^2 \\ &=\left[3x+3y\over x+y\right]^2=3^2=9 \end{aligned} $$
As a result, $f(x,y,z)$ remains constant in the domain of $\{x,y,z\in\mathbb{R}^3|x+y+z=0\}$, so its maximum value is also 9.