The problem: Find the right triangular prism of given volume and least area if the base is required to be a right triangle.
As for parameters of the right triangular prism, $V$ is volume, $A$ is surface area, $a$ and $b$ are legs of right triangle, and $h$ is the height.
Function to be minimized is then $A=ab+ah+bh+\sqrt {a^2+b^2}h$, and the constraint is $V=\frac 12abh$, where $V=constant$.
Then form the function $F=ab+ah+bh+\sqrt {a^2+b^2}h+\lambda\frac 12abh$, and calculate its gradient with each partial set to zero.
$$ (1)\frac {\partial F} {\partial a}=b+h+\frac {ah} {\sqrt {a^2+b^2}}+\lambda\frac 12bh=0, $$ $$ (2)\frac {\partial F} {\partial b}=a+h+\frac {bh} {\sqrt {a^2+b^2}}+\lambda\frac 12ah=0, $$ $$ (3)\frac {\partial F} {\partial a}=a+b+\sqrt {a^2+b^2}+\lambda\frac 12ab=0, $$
Then with the gradient of $F$ set equal to zero and the constraint, $V=\frac 12abh$, there are four equations with four unknowns($a, b, h, \lambda$).
My attempts to solve the set of four equations result in forbidding amounts of algebra. My most promising attempt involved multiplying (1) by $a$, (2) by $b$, and (3) by $h$, then adding those three equations together and solving for $\lambda$. But it was not successful.
This is my first time posting here and I hope I didn't make any mistakes with the syntax. Any help would be much appreciated so that I can continue with my self-studies.
Let's first do what you started but (1)*a +(2)*b -(3)*h. Then The first new equation will be $$ 2ab+1/2\lambda abh=0 $$ Assuming that $V>0$ we can get $a,b,h>0$. Thus we immediately obtain $$ h=-4/\lambda; $$
Now let's (1)*a-(2)*b and divide by $h$. $$ (a-b)+\frac{(a^2-b^2)}{\sqrt{a^2+b^2}}=0 $$ The last equaiton implies that or $a=b$ otherwise $a+b<0$. So from (3) we have:
EDITED $$ a(2+\sqrt{2}-2/ha)=0 $$ This implies that $h/a=\frac{2}{2+\sqrt{2}}$ and $V=\frac{a^3}{2+\sqrt{2}}$ I hope I did not make a mistake here. From here we obtam $a=b$ and $h$.