Problem: Calculate the line integral
$$ \int_{A}\frac{y\,dx-(x+1)\,dy}{x^2+y^2+2x+1} $$
where $A$ is the line $|x|+|y|=4$, travelling clockwise and making one rotation.
Answer: $-2\pi$
Solution:
The denominator $x^2+y^2+2x+1$ can be rewritten as $(x+1)^2+y^2$, which shows that there is a singularity at $(-1, 0)$.
Now, if $$ \left\{\begin{aligned} M &= \frac{y}{(x+1)^2+y^2} \\[2pt] N &= \frac{-(x+1)}{(x+1)^2+y^2} \end{aligned}\right. $$
then $$ \frac{\partial N}{\partial x} = \frac{\partial M}{\partial y} $$
Thus, Green's theorem yields:
$$ \tag{0}\int_{A-B} M\,dx + N\,dy = \iint_{D} \frac{\partial N}{\partial x} - \frac{\partial M}{\partial y}\,dx\,dy = 0 $$
$B$ can be parameterized with $$ \left\{\begin{aligned} x &= -1 + \cos t \\ y &= \sin t, \end{aligned}\right. \qquad 0 \leq t \leq 2\pi $$ which finally yields \begin{align} \int_{A} M\,dx + N\,dy &= \int_{B} M\,dx + N\,dy \\ &= \int_{0}^{2\pi} -\frac{\sin^2 t}{1} - \frac{\cos^2 t}{1}\,dt \\ &= \int_{0}^{2\pi} - 1 \,dt \\[2pt] &= -2\pi \end{align}
I don't understand how to interpret the "$A - B$" on the left-hand side of equation $(0)$. It seems to me that it does not represent a simple closed curve, as shown in the image above. How can Green's theorem then still be used?
Also, why is $B$ a unit circle around the singularity? Why not a circle of radius $2$, or a square? Why is $B$ necessary at all? I can't figure out why $B$ appears as part of the solution.
Regions in the plane and curves that bound them all have orientation. These are defined in a specific way that makes Green's Theorem work. By convention, a positively oriented closed, simple curve traces the boundary of a region with the interior of the region always on the left (and the exterior on the right). This works just fine if the region is simply connected (topologically equivalent to a disk) and so its boundary is a single curve.
However, if the region is not simply connected (it has holes), then the boundary consists of a union of multiple simple closed curves. In Green's Theorem, the integrand must be continuous on the entire region, which is why in your example, we have to remove some region around the singularity. This produces the region inside the square $A$ but outside the circle $B$.
To orient its boundary positively, we need to traverse $A$ counterclockwise and $B$ clockwise. (Check for yourself that this keeps the interior of the region always on the left of either curve.) Now, an integral over the entire boundary is an integral over $A \cup B$, but to indicate that $B$ is meant to be clockwise (its interior is actually the exterior of our region in question), we might write this as $A \cup (-B)$, or just $A - B$ for short, especially since when we integrate over a domain consisting of two disjoint pieces, we're just adding the respective integrals, the latter with a minus sign: $$ \int_{A-B} f(\mathbf{r}) \, d\mathbf{r} = \int_A f(\mathbf{r}) \, d\mathbf{r} - \int_B f(\mathbf{r}) \, d\mathbf{r}. $$
Now, your calculation actually involves the reverse orientation of the square (and hence the circle), so there is a global minus sign, but the principle still applies.