I read somewhere that if the temperatures of all points of a huge room were plotted then the derivative at a certain point would give a vector whose direction points in the direction of the hottest point in the room with its magnitude being equal to the temperature at that point.
Later on I learned that finding the grad of a scalar field(a surface) gives you a vector perpendicular to the surface(the normal). But the normal points upwards and not in the direction of the largest scalar values being generated by the scalar field. I'm confused...
Your confusion seems about the distinction between a surface level in a space where the gradient is taken and the space itself(in this case the room where the gradient of the scalar field of temperatures is plotted). The surface you mentioned in the second paragraph is a surface level of constant temperature, so the gradient must be perpendicular to it, as explained by Winther on the comments.