Let $A$ be a domain in $\mathbb R^2$ whose boundary $\gamma $ is a smooth positively oriented curve and whose area is $|A|$.
Find a function $F:\mathbb R^2\to \mathbb R$ such that $$\frac{1}{|A|}\int_\gamma Fdx+Fdy$$ is the average value of the square of the distance from the origin to a point of $A$.
I guess I should use Green's theorem at some point, but I don't know how exactly, and how to start. The distance from $(x,y)\in A$ to the origin is $\sqrt{x^2+y^2}$. So the average I believe is $\frac{\sum_{(x,y)\in A} x^2+y^2}{|A|}$? I don't know where to get from this.
For notational convenience, let me rename your domain $\Omega$, so that $\gamma = \partial \Omega$ is the positively oriented boundary of $\Omega$.
The average value of an integrable function $f : \Omega \to \mathbb{R}$ on a domain $\Omega$ is $$ \frac{1}{\lvert \Omega \rvert} \iint_\Omega f(x,y) \, \mathrm{d}A. $$ As a result, the average value over a domain $\Omega$ of the square of the distance from the origin is indeed $$ \frac{1}{\lvert \Omega \rvert}\iint_\Omega (x^2+y^2) \, \mathrm{d}A. $$ Now, if $F : \Omega \to \mathbb{R}$ is a $C^1$ function, then by Green's theorem, $$ \int_\gamma F \, \mathrm{d}x + F \, \mathrm{d}y = \iint_\Omega (F_x - F_y)\,\mathrm{d}A, $$ so your task is to reverse engineer a function $F$, such that $$ \frac{1}{\lvert \Omega \rvert}\int_\gamma F \, \mathrm{d}x + F \, \mathrm{d}y = \frac{1}{\lvert \Omega \rvert}\iint_\Omega (F_x- F_y)\,\mathrm{d}A = \frac{1}{\lvert \Omega \rvert}\iint_\Omega (x^2+y^2) \, \mathrm{d}A; $$ to do so, it suffices to reverse engineer a function $F$, such that $$ \forall (x,y) \in \Omega, \quad F_x(x,y) - F_y(x,y) = x^2 + y^2. $$ If you can't get started with that, here's an emergency hint: