The question specifies to solve this problem by maximizing $f(x,y)=xy(s-x-y)$ on an appropriate set. I admittedly do not have much of an attempt to post here. My initial thought was to try LaGrange multipliers, but I don't see how that would relate to the choice of "an appropriate set". Any advice on how to approach this is much appreciated.
Note: This is independent study material, not coursework.
On
$$xyz~\leq~\left(\frac{s}{3}\right)^3$$
where $s=x+y+z$ the you can derivate
$$\begin{align}xyz~ &\leq~\left(\frac{x+y+z}{3}\right)^3\\ \sqrt[3]{xyz}&\leq~\frac{x+y+z}{3}\end{align}$$
where in the last line the L.H.S. is the geometric mean and the R.H.S. is the arithmetic mean and so this inequality is the AM-GM inqueality and you are done.
So if $f(x,y)=xy(s-x-y)$
Then if we solve for $\frac{f(x,y)}{dx}= 0$ and $\frac {f(x,y)}{dy} = 0$ we get
$\frac{f(x,y)}{dx} = y(s - x-y) -xy = 0$ so $ys - y^2 = 2xy$. The occurs if $y=0$ or if $s-y = 2x$
$\frac{f(x,y)}{dy} = x(s-x-y) - xy = 0$ so $sx-x^2 = 2xy$. This occurs if $x=0$ or $s- x = 2y$.
As $x,y > 0$ then we have an extreme (a maximum) when $s = 2x -y = 2y-x$ or $x = y = \frac 13 s$.
So $xy(s-x-y) \le (\frac s3)^3$.