Here $\Delta$ is area of triangles, $a$ is side opposite to angle $A$ and so on
$$64 \frac{a^2b^2c^2}{16R^2}=3abc(a+b+c)$$
$$16\sin A\sin B\sin C =3(\sin A +\sin B+\sin C)$$
$$8(\cos (60)+\cos C)\sin C=3(2\cos \frac C2 ( \frac{\sqrt 3}{2})+\sin C)$$
$$8\sin C(\frac 12 +\cos C )=3(\sqrt 3 \cos \frac C2 +\sin C)$$
$$4\sin C (1+2\cos C)=3(\sqrt 3 \cos \frac C2 +\sin C)$$
$$4\sin C (4\cos ^2 \frac C2 -1)=3(\sqrt 3 \cos \frac C2 +\sin C)$$
$$8\sin \frac C2 (4\cos ^2 \frac C2 -1)=3(\sqrt 3 +2\sin \frac C2)$$
$$8(4-4\sin^2\frac C2 -1 )\sin \frac C2 =3(\sqrt 3 +2\sin \frac C2)$$
Solving the cubic
$$\sin \frac C2 =-\frac{\sqrt 3}{2}$$
So $\cos C =-\frac 12$
The given answer is $\frac 58$, but I don’t know what went wrong in my computation
Also I would like to avoid a cubic polynomial, since I used a calculator to solve this one.
Thanks
Continue with $$16\sin A \sin B \sin C= 3(\sin A+ \sin B +\sin C)$$
and simplify with $\sin A+ \sin B +\sin C= 4\cos\frac A2\cos\frac B2 \cos\frac C2$ $$32\sin\frac A2\sin\frac B2 \sin\frac C2=3 $$ or $$ 16 \left( \cos\frac{A-B}2-\sin\frac C2\right)\sin \frac C2=3 $$
which, with $A-B =60$, yields $\sin\frac C2=\frac{\sqrt3}4 $ and
$$\cos C = 1- 2\sin^2 \frac C2= \frac 58$$