I need some suggestions in the reasoning of the following problem.
I would like to find the extreme value of $$ f=xyz, \quad\text{subject to the constraint } x+y+z=a $$ To start with, I used the constraint to make the function a problem of two variables. Therefore, I had $$ f=xy(a-x-y) $$ Here onwards, I thought to use the partial derivatives to find out the critical points, i.e., $$ \begin{align} f_x=ay-2xy-y^2=0 \\ f_y=ax-2xy-x^2=0 \end{align} $$ Subtracting $f_x$ from $f_y$ I got $$ \begin{align} x^2-y^2-a(x-y)=0 \\ (x-y)(x+y-a)=0 \end{align} $$ Here I get two solutions, i.e., either $x=y$ or $x+y=a$.
If I use that $x+y=a \implies z=0$, which in turn will give us $xyz=0$, and substituting this in $f_x$, I get the solutions $(a,0,0)$, $(0,a,0)$, and $(a/3,a/3,a/3)$.
The other solutions, using $x=y$ and substituting this into $f_x$, yields $ax-2x^2-x^2=0 \implies x(a-3x)=0$, and therefore, I have two more solutions: $x=0$ and $x=a/3$. Now $x=0 \implies y=0$, and therefore, $z=a$. Thus, $(0,0,a)$ gives us the value of $xyz=0$.
$x=a/3$ will give us the same solution that we found above, i.e., $(a/3,a/3,a/3)$. Now, to test for minima and maxima $$ \begin{align} & f_{xx}=-2y \\ & f_{yy}=-2x \\ & f_{xy}=a-2x-2y \end{align} $$ and $$ \begin{matrix} pts.&(a,0,0)&(0,a,0)&(0,0,a)&(\frac{a}{3},\frac{a}{3},\frac{a}{3}) \\ f_{xx}& 0&-2a&0&-2a/3\\ f_{yy}&-2a&0&0&-2a/3 \\ f_{xy}&0&0&a&-a/3 \\ f_{xx}f_{yy}-f^2_{xy}&0&0&-a^2&a^2/3 \end{matrix} $$ Here, I can say with certainity that $(0,0,a)$ is a saddle point, which in turn implies that there are $(a,0,0)$ and $(0,a,0)$, as all these give the same value for $xyz$.
Now, considering the point $(a/3,a/3,a/3)$, according to the values it seems to be a maximum, but I believe that this is only true if $x,y,z>0$. I don't think in any of the above steps, have I made such a constraint on the domain of $x$, $y$, and $z$, and yet I get this maximum point.
Yes, your reasoning is mostly correct. You are solving the optimization problem by substitution (in comment told by Lagrange multiplier's method) with $x,y,z,a \in R$. At the end you have two cases: $a>0$ and $a<0$. If: $$1) \ \ a>0 \Rightarrow f_{xx}=f_{yy}=-\frac{2a}{3}<0,\Delta=\frac{a^2}{3}>0 \Rightarrow f\left(\frac{a}{3},\frac{a}{3},\frac{a}{3}\right) \ is \ \max.$$ $$2) \ \ a<0 \Rightarrow f_{xx}=f_{yy}=-\frac{2a}{3}>0,\Delta=\frac{a^2}{3}>0 \Rightarrow f\left(\frac{a}{3},\frac{a}{3},\frac{a}{3}\right) \ is \ \min.$$ But remember that these are just local (not global) extremes.
To simplify the analysis, consider finding extreme values of $y=3x-x^3$. $$y'=3-3x^2=0 \Rightarrow x_{1,2}=\pm1.$$ $$y''=-6x \\ y''(-1)=6>0 \Rightarrow y(-1)=-2 \ (local \ \min) \\ y''(1)=-6<0 \Rightarrow y(1)=2 \ (local \ \max).$$ Global max/min of this function do not exist, because: $$\lim_{x\to \pm\infty}(3x-x^3)=\mp\infty.$$