I've been trying to solve this problem for a while, but for too long couldn't I continue my partial solution. I would be glad if you could shed some light on my solution.
The task: Given the vector field $\vec{F}(x,y)\equiv(P(x,y),Q(x,y))$ such that $P(x,y)=y^3-3y+xy^2$ and $Q(x,y)=3x-x^3+x^2y$, which is defined on $D=\{(x,y)\ |\ x^2+y^2\leq2\}$, find a simple and smooth curve $C$ from $A(1,1)$ to $B(-1,-1)$ which is inside $D$ such that:
$$\int_C\vec{F}\cdot d\vec{r}$$
gets its maximum value.
My solution
Let $C$ and $C_0$ be two curves that satisfy the requirements of the question, such that they are the boundaries of a closed area $S\subseteq D$. We will choose $C$ to be positively oriented, whereas $C_0$ will be negatively oriented.
Now, as $\vec{F}\in C^1$, we are able to use Green's Theorem. We can see that:
$$\frac{\partial Q}{\partial x}-\frac{\partial P}{\partial y}=6-3(x^2+y^2)$$ So according to the theorem, defining $\Gamma\equiv C\ \cup\ C_0$:
$$\oint_{\Gamma}\vec{F}\cdot d\vec{r}=\int_C\vec{F}\cdot d\vec{r}-\int_{C_0}\vec{F}\cdot d\vec{r}=\iint_Sr(6-3r^2)drd\theta$$
Accordingly:
$$\int_C\vec{F}\cdot d\vec{r}=\int_{C_0}\vec{F}\cdot d\vec{r}+\iint_Sr(6-3r^2)drd\theta$$
Since $\displaystyle \iint_Sr(6-3r^2)drd\theta$ is always positive inside $D$ $(0\leq r\leq\sqrt2)$, the obvious would be to choose $C_0$ such that $S$ is the biggest area we can fit inside $D$. The problem is, that this might cause $\displaystyle \int_{C_0}\vec{F}\cdot d\vec{r}$ to be relatively small.
That is where I got stuck.
P.S.: I also looked at the field, geometrically. It seems that inside $r=\sqrt2$ it looks different than the outside. I bet that's a thing I should take into account, but I don't know how.
Thanks!
Without loss of generality, consider a fixed curve $C_0$. Specifically, take $C_0$ to be the straight line from $B$ to $A$.
As you noted, $\frac{\partial Q}{\partial x} - \frac{\partial P}{\partial y}$ is non-negative over the domain. Thus, maximizing $\iint_S (\frac{\partial Q}{\partial x} - \frac{\partial P}{\partial y})dA$ is a matter of taking the maximal area "above" the line parameterized. In particular, if there is a $C$ such that the interior region $S$ that results from appending $C$ to $C_0$ is $S = \{(x,y) \in D: y \geq x\}$, then it must be that $\int_C \vec F \cdot d \vec r$ is maximal.
Thus, the best we could do is $C = \{(x,y): x^2 + y^2 = 2, y \geq x\}$.