I'm learning calculus but I'm very stuck in this exercise.
I know that the volume is : 2HL + 2HW + LW (only LW because open top).
Thanks to the exercise we know that:
W = 12 - 2x L = 20 - 2x H = x
Consequently I replaced the values:
2x(20 - 2x) + 2x(12 - 2x) + (20 - 2x)*(12 - 2x)
= -4x^2 + 240
My function is f(x) = -4x^2 + 240
However, I don't understand why in the correction it is written 4x^3 - 64x^2 +240x ?

You haven't calculated the volume of the box, but the surface of the cutout.