I don't think I quite understand how to go about this.
My solution so far:
$\oint_C |z|^2 dz = \oint_C (x^2 + y^2)dz = \oint_C (x^2 + y^2) d(x+iy) = \oint_C x^2 + y^2 dx + i\oint_Cx^2+y^2dy$.
Then, I just plug in $(0,0) \to (1,0), (1,0) \to (1,1), (1,1) \to (0,1), (0,1) \to (0,0)$.
For $(0,0) \to (1,0)$, I get $\frac{1}{3}$
For $(1,0) \to (1,1)$, I get $1 + i\frac{4}{3}$
For $(1,1) \to (0,1)$, I get $-\frac{4}{3} - i$
For $(0,1) \to (0,0)$, I get $-i\frac{1}{3}$
When all is said and done, I get $0$, but apparently the answer is $-1+i$. I'm not sure where I went wrong, or if I just don't understand what I'm doing.
You're kind of missing the point with your treatment of the contour integral. All you need to do is evaluate separately on each side. To do so, you need to parametrize. Let the sides of the square be $C_1$, $C_2$, C_3$, and $C_4$, where these sides are defined as follows:
$$C_1 = \{z : z=x , x\in [0,1] \}$$ $$C_2 = \{z : z=1+i y , y\in [0,1] \}$$ $$C_3 = \{z : z=x+i , x\in [1,0] \}$$ $$C_4 = \{z : z=i y , y\in [1,0] \}$$
so that
$$\oint_C dz \, |z|^2 = \int_0^1 dx \, x^2 + i \int_0^1 dy (1+y^2) + \int_1^0 dx (1+x^2) + i \int_1^0 dy \, y^2$$
Note that the integrals over $x^2$ and $y^2$ cancel, but we are left with
$$\oint_C dz \, |z|^2 = -1+i$$