Here is an image of the problem...
So labeled the bottom base as $x$ and the upper base as $y$ this helped me come up with two equations, one for perimeter $P$ and one for area $A$:
$$P = 24 = x + 2\frac{(y-x)}{2cos\theta}$$
and
$$A = \frac{x+y}{2} \times \frac{y-x}{2}tan\theta$$
I did some simplifying but I am stuck on which variable I should isolate and from which equation, or if maybe I've made a mistake? Answer in the book is $\theta = \pi/3 $ and the legs of the trapezoid $= 8$ cm
On
The gutter is formed from the large strip of metal by bending a smaller strip upward on either side of the larger strip. Let the width of the smaller strip be $r.$
The trapezoidal cross-section of the gutter can be divided into a rectangle of width $$x = 24 - 2r \tag1$$ between two right triangles with hypotenuse $r$ and legs $r\cos\theta$ and $r\sin\theta.$ Then the upper base of the trapezoid is $$y = x + 2r\cos\theta.\tag2$$ From $(2)$ we get $$2r = \frac{y-x}{\cos\theta},\tag3$$ and then from $(1)$ we get $$24 = x + 2r = x + \frac{y-x}{\cos\theta}.\tag4$$ This is the same as your "perimeter" equation. (You were wrong to set $P=24$ if $P$ was meant to be the perimeter; the perimeter is actually $24+y.$ But this is irrelevant to the problem, since you have no reason to make any further use of the symbol $P.$)
The height of the trapezoid is $h = r\sin\theta.$ Using $(3)$ to substitute for $r,$ we have $$h = \frac{y-x}{2\cos\theta}\sin\theta = \frac{y-x}{2}\tan\theta.$$ The area of the trapezoid is therefore $$A = \frac{x+y}{2}h = \frac{x+y}{2} \left(\frac{y-x}{2}\tan\theta\right).$$ Again that agrees with what you found.
You can use $(4)$ to eliminate one variable from the equation for area: $\frac{24-x}{y-x} = \frac{1}{\cos\theta},$ so $$\left(\frac{24-x}{y-x}\right)^2 = \sec^2\theta = 1 + \tan^2\theta.$$ Therefore $$A = \frac{x+y}{2} \left(\frac{y-x}{2}\right) \sqrt{\left(\frac{24-x}{y-x}\right)^2 - 1}.$$ You can further simplify this by maximizing $A^2$ instead of $A$ (to eliminate the square root) and perhaps substituting a new variable for $y - x.$
Or you could start over. From $(2),$ $y = 24 - 2r + 2r\cos\theta.$ From basic trigonometry, $h = r\sin\theta = r\sqrt{1 - \cos^2\theta}.$ The area of the trapezoid is therefore $$A = \tfrac12(x+y)h = (12 - r + r\cos\theta)r\sqrt{1 - \cos^2\theta}.\tag5$$ You can then substitute a new variable name for $\cos\theta$ and optimize. Alternatively, since $r\cos\theta = \sqrt{r^2 - r^2\sin^2\theta} = \sqrt{r^2 - h^2},$ $$A = \tfrac12(x+y)h = (12 - r + \sqrt{r^2 - h^2})h.\tag6$$ I think $(5)$ looks easier to work with than $(6),$ since you can just square both sides to eliminate the square root, but you can also take partial derivatives of the right-hand side of $(6)$ directly.
Denote by $y$ the amount bended up on both sides and by $x$ the remaining base. Then $x+2y=24$ and $$A=(x+y\cos\theta) y\sin\theta=\bigl(24-y(2-\cos\theta)\bigr)y\sin\theta=: f(y,\theta)\ .$$ One computes $$\eqalign{f_y&=2\sin\theta(12-2y+y\cos\theta)\ ,\cr f_\theta&=y\bigl(2\cos^2\theta+2\cos\theta(12-y)-1\bigr)\ .\cr}$$ Put $\cos\theta=:u$. From $f_y=0$ we get $y={12\over2-u}$, and $f_\theta=0$ then leads to $u={1\over2}$. We therefore have a stationary point at $y=8$, $\theta={\pi\over3}$.
This means that the optimal gutter is half of a regular hexagon.