Let $X$ be a compact complex surface.
This definition is from Donaldson, Kronheimer: The Geometry of 4-Manifolds, p. 209:
Definition: A holomorphic $SL(2,\mathbb{C})$ bundle $E$ over $X$ is called stable if the following holds: For each line bundle $L$ over $X$ we have $$h^0(Hom(L,E)) \neq 0 \Rightarrow deg (L)<0.$$
Denote by $K_X$ the canonical bundle of $X$, let $F$ be the ideal sheaf of a finite set of points in $X$. Let $E$ be a bundle over $X$ fitting into the following short exact sequence:
$$0 \rightarrow \mathcal{O}_X \rightarrow E \rightarrow K_X \otimes F \rightarrow 0.$$
The following lemma is a consequence of Lemma 10.3.7 in Donaldson, Kronheimer: The Geometry of 4-Manifolds:
Lemma: Let $X$ be such that $Pic(X)=\langle K_X \rangle$, $deg(K_X)>0$. Then $E$ is stable.
In the book, the proof goes by showing that $h^0(E \otimes K_X^{-1})=0$.
Question 1: How can I see that the definition of stability given above is equivalent to the usual definition of slope stability, i.e.: every subbundle $E'$ shall have strictly smaller than $E$?
Question 2: $h^0(Hom(L,E)) \neq 0 \Rightarrow deg (L)<0$ is equivalent to $deg(L)\geq 0 \Rightarrow h^0(Hom(L,E)) = 0$. All line bundles with positive degree on $X$ are powers of $K_X$, so we must check that $h^0(Hom(K_X,E))=h^0(E \otimes K_X^{-1})=0$ and $h^0(Hom(\mathcal{O}_X,E))=0$. Why is the second condition not checked in the proof in the book? (Also it doesn't seem true, which can be seen if I take global sections in the short exact sequence above) How does stability follow from $h^0(E \otimes K_X^{-1})=0$?
Q1: First, in algebraic geometry (and other fields), subbundles have a different meaning than what you assert. A subbundle $E'\subset E$, where $E',E$ are bundles mean the quotient $E/E'$ is a bundle. This works well for non-singular curves, but not in higher dimension. So, the slope stability for higher dimensional situation is, given any proper subsheaf $E'\subset E$, $\deg E'<\deg E$. In the situation above (your quotation of definition), the bundle in question has degree zero (being an $SL(2,\mathbb{C})$ bundle) and rank two. Staring with your definition, let $L\subset E$ a proper subsheaf. I will let you convince yourself that the only case of interest is when rank of $L=1$. The fact that $L\subset E$ implies, $h^0(L^{-1}\otimes E)\neq 0$ and thus $\deg L<0$. The converse is equally straight forward.
Q2: Why do you think one should check $h^0(\operatorname{Hom}(\mathcal{O}_X,E)\neq 0$, since it is clearly false from your exact sequence?