One of my favorite ways of thinking about the derivative of a function is as something that provides the best linear (or, well, affine) approximation of the function near a point. So, near a point $x_0$, $f(x)\approx f(x_0)+f'(x_0)(x-x_0)$. So this got me thinking if we could use this property to actually define the derivative. It would be as follows:
Let $y(x)=f(x_0)+m(x-x_0)$. We say that $m$ is the derivative of $f$ at $x_0$ if, for any straight line $\tilde y(x)$, there is a $\delta$ such that $|y(x)-f(x)|\leq|\tilde y(x)-f(x)|$ for all $x \in (x_0-\delta,x_0+\delta)$ (except maybe $x_0$ itself?).
Is this actually a reasonable way to define the derivative? Is this equivalent to the usual definition? I tried to prove it is, but I'm not that good at working with these kinds of epsilon-delta definitions. My intuition tells me that, at the very least, the derivative should have this property, but is this property all we need?
You are on the track to one common definition of the derivative that agrees with the standard limit definition, and one that neatly generalizes to higher dimensions, once you learn a little linear algebra (by making $m$ a suitable linear transformation rather than just a number). However, as presented, it is not quite there.
Your idea that we should find the tangent line, the line that "best approximates" $f$ "close to" $x_0$, is entirely right. But we have to be careful how we phrase "best approximates" and "close to".
To illustrate, consider, for instance, $f(x)=x^2$ and $x_0=1$. We know we are after the line $y(x)=2x-1$, so we establish that. Now pick a $\delta>0$. Look at the graphs of $f$ and $y$ on the interval $(1-\delta, 1+\delta)$. (If you're actually making a drawing, I suggest picking $\delta=1$ or something, any $\delta$ will have the same problems, and this way you can actually see it.) There are certainly many lines $\tilde y$ that lie between $f$ and $y$ at least somewhere on that interval.
So you need to do something a little different. The standard approach is to quantify how good the approximation is as $\delta$ decreases. And that way, it turns out there will be at most a single line that does the job. This is how it's usually done:
Require the error in approximation, which is to say the quantity $|f(x)-y(x)|$, to be such that $$ \frac{\displaystyle\sup_{x\in(x_0-\delta,x_0+\delta)}|f(x)-y(x)|}{\delta} $$ goes to $0$ as $\delta$ goes to $0$. In other words, the largest approximation error on the interval $(x_0-\delta,x_0+\delta)$ shrinks significantly faster than $\delta$ itself as $\delta$ gets smaller.