What is the maximum possible area that a quadrilateral can have, if the lengths of three of its sides are given as 3, 4 and 5, while the fourth side can have arbitrary length? (Thinking of it as three fence segments arranged around a straight wall such that the area becomes maximal)
My approach: Since for given lengths a cyclic quadrilateral will have maximum area, I started out from here. According to Heron’s theorem the squared area of a cyclic quadrilateral equals $A^2 = (s-5)(s-4)(s-3)(s-2x)$ with $s = (5+4+3+2x)/2$ equaling half of the circumference and $x$ half of the unknown side. Rearrangement leads to $A^2=(3+x)(2+x)(1+x)(6-x)$. Setting the x-derivative to zero yields the maximum area $A = 20.495$ with $x = 4.0279$ (and the missing side $2x = 8.0558$). Is this correct? Is the maximal quadrilateral at all cyclic as I assumed? And If yes, isn’t there a more elegant and intuitive geometric solution?
Your approach is fine and correct. If four segments $a_1,a_2,a_3,a_4$ satisfy the inequalities $$ a_i<\frac12\sum_i a_i,\tag1 $$ which is necessary for construction of any quadrilateral, it is possible to construct a cyclic quadrilateral out of these segments as well. There are up to congruence two such quadrilaterals, both having the same area, which is by the Bretschneider's formula larger than the area of any other quadrilateral constructible of the same sides.
Though the solution of your problem admits a simple geometric interpretation (the fourth side is the diameter of the circumscribed circle), the construction of the side by geometric means (i.e. with compass and unmarked straightedge) is impossible.
Indeed the length in question is the (only) positive solution of the cubic equation: $$ x^3-(a^2+b^2+c^2)x-2abc=0\tag2 $$ with integer values $a,b,c$.
By the well-known Lemma the roots of the equation $(2)$ are constructible if and only if at least one of them is rational.
Assume the equation $(2)$ has a rational root $x=\frac pq$, with $p,q\in\mathbb Z, (p,q)=1, q>0$. Substituting the value into $(2)$ one obtains: $$p^3=[(a^2+b^2+c^2)pq+2abc q^2]q\implies q\mid p\implies q=1.$$
On the other hand the same equation can be rewritten as: $$ [p^2-(a^2+b^2+c^2)q^2]p=2abcq^3\implies p\mid 2abc. $$
The direct substitution of all 32 (including negative) divisors of $2abc=120$ into $(2)$ verifies that none of them satisfies the equation. This proves that the fourth side of the quadrilateral maximizing its area is not constructible.