I was given the 1-form $\omega = \sqrt{xy} dx+x^2y^2dy$ and was asked to integrate it along the triangle with vertices $(0,0),(1,0),(1,1)$ with anticlockwise orientation.
First, I did it with the definition, parameterizing the segments with $\gamma_1(t)=(1,t),\gamma_2(t)=(1-t,1-t),\gamma_3(t)=(0,t)$ with $t \in [0,1]$ and got
$$\int_{\gamma_1} \omega=\int_0^1t^2dt=\frac{1}{3}, \; \; \int_{\gamma_2} \omega=\int_0^1(t-1-(t-1)^4)dt=-1+\frac{1}{2}+\frac{1}{5}=-\frac{3}{10},\; \; \int_{\gamma_3} \omega=\int_0^10dt=0$$ So the total value should be the sum: $\frac{1}{30}$
Then I thought it would be nice to use Green's formula to practice and to check the result, so if we denote the triangle by $\Delta$: $$\int_{\partial \Delta }\sqrt{xy} dx+x^2y^2dy= \iint_\Delta \left( \frac{\partial}{\partial x}(x^2y^2)-\frac{\partial}{\partial y}(\sqrt{xy})\right)dxdy= \iint_\Delta \left( 2xy^2-\frac{1}{2}\sqrt{\frac{x}{y}}\right)dxdy$$ $$=\int_0^1\left( \int_0^x \left( 2xy^2-\frac{1}{2}\sqrt{\frac{x}{y}}\right)dy\right)dx= \int_0^1\left( \frac{2}{3}x^4-x \right)dx=-\frac{11}{30}$$ So something clearly went wrong. I've checked my calculations many times and showed a couple of friends, we found nothing.
I also checked the hypothesis of Green's theorem and noticed that the partial derivatives of $\sqrt{xy},x^2y^2$ must be continuous over the whole triangle, which doesn't hold when $x=0$ or $y=0$. However, the derivatives are Riemann integrable (and Lebesgue integrable) over the interior of the triangle and I thought there wouldn't be any problem. I have gone over the proof of the theorem in elemental domains and don't see why the formula wouldn't work.
Your mistake is in line integral along $\gamma_2$ from $(1, 1)$ to $(0, 0)$.
$\displaystyle \int_{\gamma_2} \omega = \int_0^1 [(t-1)-(t-1)^4] \ dt = \bigg[\frac{(t-1)^2}{2} - \frac{(t-1)^5}{5}\bigg]_{t=0}^{t=1} = - \frac{1}{2} - \frac{1}{5}$
$ = \displaystyle -\frac{7}{10}$
So the line integral along the edges of the triangle adds up to
$\displaystyle \frac{1}{3} - \frac{7}{10} = - \frac{11}{30}$
The same result you get using Green's theorem.