Define $$\oint_C f(z) \overline{dz}= \overline{\oint_C \overline{f(z)}dz}\;.$$ If $P(z)$ is a polynomial and $C$ denotes the circle $|z-a|=R$ (counter-clockwise), show that $$\oint_C P(z) \overline{dz}=-2\pi iR^2P'(a)\;.$$
Is the problem have any thing to do with cauchy's integral formula? thank you.
Yes, although you couldn't see that from the definition you have. I recommend that you use $(z-a)(\bar z-\bar a)=R^2$ to get $d\bar z = -\dfrac{R^2}{(z-a)^2}\, dz$.