We have an equilateral triangle and want to inscribe a square, in such way that maximizes the area of the square.
I sketched two possible ways, not to scale and not perfect.
Note I am not sure if the second way will really have all square corners touching the triangle sides.
The second case appears to have bigger side-lengths of the square, so bigger area. But I do not know how to determine the angles involved. How to solve this?


The second configuration (square has edge contact with triangle) indeed has a bigger inscribed square. If the square has unit sides, the triangle's side is $1+\frac2{\sqrt3}$:
The symmetric first configuration may be resolved as follows. Set the unit square's bottom corner as $(0,0)$, so that the top corner is $(0,\sqrt2)$. Let the side length of the triangle be $r$. Then we have, by similar triangles, $$\frac{(\sqrt3/2)r-\sqrt2/2}{\sqrt2/2}=\sqrt3$$ $$r=(1+\sqrt3)\sqrt{\frac23}=2.230\dots$$ and this is greater than $1+\frac2{\sqrt3}=2.154\dots$, so the first configuration has a smaller inscribed square than the second.