I'm sorry if I sound too ignorant, I don't have a high level of knowledge in math.
The function $f(z)=z^2$ (where $z$ is a complex number) has a derivative equal to $2z$.
I'm really confused about this. If we define the derivative of $f(z)$ as the limit as $h$ approaches $0$ (being $h$ a complex number) of $(f(z+h)-f(z))/h$, then clearly the derivative is $2z$, but what does this derivative represent??
Also, shouldn't we be able to represent a complex function in 4-dimensional space, since our input and output have 2 variables each ($z=x+iy$) and then we could take directional derivatives...right?
But if we define the derivative as above, it would be the same if we approach it from all directions. That's what's bothering me so much.
I would really appreciate any explanation. Thanks!


That's precisely what we do.
Well, it represents what we have defined: the limit of the incremental ratio, same as in the real case. Probably you're wondering if we can interpret this complex derivative geometrically or visually, as we interpret the (real) derivative as the slope of the tangent line... There is no such a simple pictorial interpretation.
Indeed, you could consider each of the two components (real and imaginary) separatedly, both for the variable and for the function, and then you'd get four (real) derivatives. And, yes, it's natural to ask how these four derivatives are related to the complex derivative, and if there are some (necessary and/or sufficient) restrictions on them so that the complex derivative gives the same value no matter the "direction" (as one would want)... Behold the Cauchy–Riemann equations and holomorphic functions.