Let $F:V \rightarrow W$, where $V,W$ are two vector spaces (may not be finite dimensional). Suppose $F$ is Lipschitz continuous with Lipschitz constant $C$ that is $\lVert F(\alpha + \xi) - F(\alpha)\rVert \leq C \lVert \xi \rVert $ for all $\alpha , \xi \in V$. Now I want to show that if $F$ is differentiable at $\alpha$ then $ \lVert dF_{\alpha} \rVert \leq C$ where $dF_{\alpha}$ is the derivative of $F$ at $\alpha$.
Here is what I have done so far: $F$ is differentiable at $\alpha$ means that there exists a linear approximation to the difference operator which we call $dF_{\alpha}$ i.e., $$ F(\alpha + \xi)-F(\alpha) = dF_{\alpha}(\xi) + \mathcal{o}(\xi) $$ where $\mathcal{o}(\xi)$ is the error term and $\left \lVert \frac{\mathcal{o}(\xi)}{\xi} \right \rVert \rightarrow 0$ as $\xi \rightarrow 0$. So $\lVert dF_{\alpha}(\xi) + \mathcal{o}(\xi) \rVert = \lVert F(\alpha + \xi)- F(\alpha)\rVert \leq C \lVert \xi \rVert$. This gives us that $\lVert dF_{\alpha}\rVert = \sup_{\xi\neq 0} \left \lVert \frac{dF_{\alpha}(\xi)}{\xi} \right \rVert \leq C+ sup_{\xi \neq 0}\frac{\lVert \mathcal{o}(\xi)\rVert}{\lVert \xi \rVert}.$ Now I am not sure why the second term here needs to be zero.
Your argument up to $\lVert dF_{\alpha}(\xi) + \mathcal{o}(\xi) \rVert \leq C \lVert \xi \rVert$ is correct. The subsequent formula needs work, because (a) one can't divide by a vector; (b) we don't need all nonzero vectors, just all vectors of a given norm. That is, for every $r>0$ we have $$ \lVert dF_{\alpha}\rVert = \frac{1}{r}\sup_{\|\xi\|=r } \| dF_{\alpha}(\xi)\| \le C + \frac{o(r)}{r} = C+o(1) $$ as $r\to 0$. Since $\lVert dF_{\alpha}\rVert$ does not depend on $r$, the $o(1)$ term can be ignored: $\lVert dF_{\alpha}\rVert \le C$.
(Formally: if $\lVert dF_{\alpha}\rVert$ were greater than $C$, let $\epsilon = \lVert dF_{\alpha}\rVert - C$; then there is $r>0$ such that the term we called $o(1)$ is less than $\epsilon$, a contradiction. But since you are using $o$-notation already, I assume you don't want to write out all the epsilons.)