Given a scalene triangle $ABC$ and an inscribed equilateral triangle whose vertices lie on different sides of $\triangle ABC$, what is the maximal ratio of the area of the equilateral triangle to that of the original triangle?
I would expect an answer as a ratio of polynomials in sines and cosines of the angles of $\triangle ABC$. I got such an expression via a clunky unsymmetrical method, but it but it was so messy that I gave up trying to simplify it. However, a more intelligent method might well yield a formula of reasonable length.


There isn't a single inscribed equilateral triangle: by choosing a point on a side, rotating another side by $60^\circ$ around that point and marking the intersection point with the third side, we always get an inscribed equilateral triangle.
If you mean what is the maximum ratio $\frac{[ABC]}{[DEF]}$, the answer is given by a rescaled Napoleon triangle. We may notice that such ratio is also the ratio between the areas of the largest circumscribed equilateral triangle and the original triangle. On the ther hand, it is quite simple to describe the set of the circumscribed equilateral triangles: let we consider the Napoleon/Torricelli/Fermat configuration:
We have $O_B O_C\perp AV_A$ and so on. If we take a point $P$ on $\Gamma_A$ (the circumcircle of $BCV_A$), draw a line through $B$ till meeting $\Gamma_C$ at a new point $Q$, then draw a line through $A$ till meeting $\Gamma_B$ at a third point, the (grey) triangle defined this way is equilateral. It follows that we simply need the greatest $PQ$ segment among the segments built through the previous procedure. This segment has to be perpendicular to the radical axis of $\Gamma_A$ and $\Gamma_C$, so the Napoleon triangle of $ABC$ is the smallest inscribed equilateral triangle in $A'B'C'$. By rescaling, we get the solution. It is interesting to point out that the side length of $O_A O_B O_C$ is $\frac{1}{\sqrt{3}}AV_A$, and $AV_A=BV_B=CV_C$ is the length of the Steiner net of $ABC$, i.e. $FA+FB+FC$, with $F\in\Gamma_A\cap\Gamma_B\cap\Gamma_C$ being the Fermat point of $ABC$.