I got confused with a very simple problem.
Suppose we have a function $f(x_1,x_2)=3x_1+5x_2$
The gradient of $f$ is $(3,5)^T$ at any point.
So, if I evaluate $f'(5,1)$, it will be $(3,5)^T$ at it is not perpendicular to contour lines. What was wrong here?
A vector is not a place. It is a motion. The vector $(3,5)$ is not an arrow that ends at the place with coordinates $(3,5)$. It is an arrow that starts somewhere and moves 3 units to the right and 5 units up. If you draw it starting at $(5,1)$, it will end at $(5+3,1+5) = (8,6)$, and if you draw it this way you will see that it is perpendicular to the contour line.
The arrow you drew starts at $(3,5)$ and ends at $(5,1)$, and represents the vector $(3-5, 5-1) = (-2,4)$. It's the wrong arrow.