If you look at a function "infinitely close", the difference between two points is a line:
__
__/
__/
Where each "__" is a point, and "/" is the value of a derivative (assume the two "/"s have different slopes). I have this intuition because a function can be approximated by a line at an infinitely close distance (i.e. "Linear Approximations")
If you use the above graph of the function to graph the derivative, the derivative looks like this:
_
_|
So at an infinitely close distance the derivative looks like the second graph above.
But now if you look at the derivative at an infinitely close distance, it looks the the first graph. So how can the derivative look two different ways at an infinitely close distance? It looks like the first graph when you look at it directly at an infinitely close distance, but the second graph if you look at its antiderivative at an infinitely close distance and use that to plot it.
I know this obviously isn't rigourous but what part of my intuition is wrong?
Functions which look like straight lines when you zoom in are called differentiable functions. Not all functions are differentiable. The function $x \mapsto |x|$ is not differentiable at $x=0$. The graph $y=|x|$ looks like a $\vee$ and, no matter how much you zoom in on $x=0$, it will always look like a $\vee$.
If I think I understand your question then you are thinking of a graph as lots of points which are connected to one another by straight line segments. The below diagram shows $y=\sin x$.
Each one of these straight line segments will have its own slope/gradient. Then we can plot the slopes of these straight line segments on another graph. This will give a "step function":
The point is that we need to look what happens as the points get closer and closer together. In the case of the graph $y=\sin x$ you will have more and more dots and more and more line segments.
The slopes/gradients of the line segments will hold their values for less time (because the line segments are shorter) and the difference between the steps will reduce because the line segments change less from one to the next.
In the end, the points will all come together so that the line segments shrink down to points and we will get the graph $y=\sin x$ back:
In terms of the gradient graph, the height between the steps will shrink to zero and the length of each step will shrink to zero so that we again get lots of points making the curve of the derivative (which happends to be $\tfrac{\mathrm{d}y}{\mathrm{d}x}=\cos x$)