When we say least square method for linear functions, then we mean that for each point in our plot there is a corresponding square which is just touches the linear function (see picture below)
My question is: Is this the same for an exponential function? I mean, if we had a graph with exponential slope, would we also in theory make squares as on the picture above?
Squared residuals measure the distance between your data set and a data set with the same x coordinates modeled perfectly by your regression line/curve (as vectors in n-dimensional space).
That is to say, the mathematics of least squares works out very nicely and can be used for all sorts of types of regression: linear, quadratic, exponential, logarithmic, sinusoidal, etc. For any sort of collection of curves you can ask "what curve will give the smallest total area of these squares between the data and my curve?"
This will give you different answers than Semiclassical's suggestion in the comments of taking logs and doing a linear least squares. Taking logs will make it more important for your regression exponential to do well for values close to 0 (since their logs will be farther apart). Depending on your circumstances, this may actually be preferred.