I'm trying to compute the following surface integral:
$\int\int_{O} y dy \wedge dz + z dz \wedge dx + x dx \wedge dy$,
where
$O= \lbrace (x,y,z) \in \mathbb{R^3} : x^2+y^2 \leq 4 \wedge z= x^2-y^2 \rbrace $.
The orientation is to be chosen such that the normal vector has a positive z-coordinate in the origin. My problem is, that if I want to apply the divergence theorem, I get zero:
$K = (y,z,x) \Rightarrow div(K)=0$ and so $\int\int_{O} y dy \wedge dz + z dz \wedge dx + x dx \wedge dy = \int\int_{O'} div(K) dV$, but I think I'm doing something wrong.
You are definitely doing something wrong, but you still get the right answer. I find the notation given for a surface integral pretty much unreadable but I think that you mean given the vector field $\vec K=\langle y,z,x\rangle$ you want the upward flux $$\int\int\vec K\cdot d^2\vec A$$across the surface $z=x^2-y^2$, $x^2+y^2\le4$. You are applying the divergence theorem but the hypothesis of that theorem requires a closed surface, and the surface in question isn't closed. Repeat the exercise with $\vec K=\langle0,0,1\rangle$ and you won't be so lucky.
First method is simply to parameterize the surface directly: along the surface, $\vec r=\langle x,y,z\rangle=\langle r\cos\theta,r\sin\theta,r^2\cos2\theta\rangle$, so $$d\vec r=\langle\cos\theta,\sin\theta,2r\cos2\theta\rangle dr+\langle-r\sin\theta,r\cos\theta,-2r^2\sin2\theta\rangle d\theta$$ and $$\begin{align}d^2\vec A &=\pm\langle\cos\theta,\sin\theta,2r\cos2\theta\rangle dr\times\langle-r\sin\theta,r\cos\theta,-2r^2\sin2\theta\rangle d\theta\\ &=\pm\langle-2r^2\cos\theta,2r^2\sin\theta,r\rangle drd\theta\\ &=\langle-2r^2\cos\theta,2r^2\sin\theta,r\rangle drd\theta\end{align}$$ because we want the upward flux. Then $\vec K=\langle r\sin\theta,r^2\cos2\theta,r\cos\theta\rangle$ and $$\int\int\vec K\cdot d^2\vec A=\int_0^{2\pi}\int_0^2\langle r\sin\theta,r^2\cos2\theta,r\cos\theta\rangle\cdot\langle-2r^2\cos\theta,2r^2\sin\theta,r\rangle drd\theta=0$$ because $$\int_0^{2\pi}\sin\theta\cos\theta d\theta=\int_0^{2\pi}\sin\theta\cos2\theta d\theta=\int_0^{2\pi}\cos\theta\times1 d\theta=0$$ Another way is to view the problem through the lens of Stokes' theorem: let $\vec F=\langle\frac12z^2,\frac12x^2\frac12y^2\rangle$. Then $K=\vec{\nabla}\times\vec F$ and we have $$\int\int\vec K\cdot d^2\vec A=\int\int\vec{\nabla}\times\vec F\cdot d^2\vec A=\oint\vec F\cdot d\vec r$$ Where the line integral is taken around the boundary of the surface. Along the boundary we may parameterize $\vec r=\langle x,y,z\rangle=\langle 2\cos\theta,2\sin\theta,4\cos2\theta\rangle$. Then $$d\vec r= \langle-2\sin\theta,2\cos\theta,-8\sin2\theta\rangle d\theta$$ and $$\vec F=\langle8\cos^22\theta,2\cos^2\theta,2\sin^2\theta\rangle=\langle4+4\cos4\theta,1+\cos2\theta,1-\cos2\theta\rangle$$ so $$\oint\vec F\cdot d\vec r=\int_0^{2\pi}\langle4+4\cos4\theta,1+\cos2\theta,1-\cos2\theta\rangle\cdot\langle-2\sin\theta,2\cos\theta,-8\sin2\theta\rangle d\theta=0$$ because $$\begin{align}\int_0^{2\pi}\sin\theta d\theta &=\int_0^{2\pi}\sin\theta\cos4\theta d\theta=\int_0^{2\pi}\cos\theta d\theta \\ &=\int_0^{2\pi}\cos\theta\cos2\theta d\theta=\int_0^{2\pi}\sin2\theta d\theta\\ &=\int_0^{2\pi}\sin2\theta\cos2\theta d\theta=0\end{align}$$ So the integral works out to zero by both right methods as well as the wrong method :)