By the existence of derivative, we mean that the left hand and right hand derivatives are equal. If they're not equal then the derivative doesn't exist and everything else is useless. But I think that the left hand and right hand derivatives are the actual things which are required to exist for approximating functions irrespective of whether they're equal or not.
For example, if the right hand derivative of a function exists and is $f'(x)$, and if $x>a$, then we don't need the derivative to exist to write this:
$$f(x)\approx f(a)+f'(a)(x-a)$$
I think if $x>a$, then approximating $f(x)$ should only require the right hand derivative. And for $x<a$, only the left hand derivative is required to approximate $f(x)$. We don't need them to be equal.
This leads to the notion of approximation from below and approximation from above. However, the usual convergence-to-the-correct-value properties that Taylor polynomials have cease to hold, as do the usual error bounds.
So you can do it, but the whole mathematical theory of Taylor polynomials stops applying. Furthermore, I believe that Taylor's Theorem doesn't hold. At which point you should probably stop calling them Taylor polynomials