Which one is the $W^{3,2}$-norm of a function $u(x,y)$?
A: $(\int |u|^3+(u_x^2+u_y^2)^{3/2}+(u_{xx}^2+u_{yy}^2+u_{xy}^2+u_{yx}^2)^{3/2} \, dx \, dy)^{1/3}$?
B: $(\int |u|^3+|u|_{x}^3+|u|_{y}^3+|u|_{xy}^3+|u|_{yx}^3+|u|_{yy}^3+|u|_{xx}^3 \, dx \, dy)^{1/3}$
I have been confused by the notation of my book. My book uses $|Du|$ to denote $(u_x^2+u_y^2）^{1/2}$. I am not $100\%$ sure which one is correct.
Note that you are working in $W^{k,p}$ with $p=3$, so in $W^{2,3}$ and not $W^{3,2}$.
The second one (but putting the derivative indices within the absolute value) is the correct Sobolev norm. We have
$$\|u\|_{W^{2,3}}=\left(\sum_{|\alpha|\leq2}\int|D^\alpha u|^3\,dx\,dy\right)^\frac{1}{3}=\left(\int\sum_{|\alpha|\leq2}|D^\alpha u|^3\,dx\,dy\right)^\frac{1}{3}=\left(\int|u|^3+|u_{x}|^3+|u_{y}|^3+|u_{xy}|^3+|u_{yx}|^3+|u_{yy}|^3+|u_{xx}|^3 \, dx \, dy\right)^{1/3}.$$
Please note that when we have $\alpha=(0,2)$, we mean $D^\alpha u=u_{yy}$. In the same way $D^{(0,0)} u=u$, $D^{(1,0)} u=u_{x}$, $D^{(1,1)} u=u_{xy}$, and so on, as long as we have $|\alpha|\leq 2$ (the sum of vector elements is at most $2$). This is very different from $$|D^ku|=\left(\sum_{|\alpha|=k}|D^\alpha u|^2\right)^\frac{1}{2},$$ where $k$ is an integer and not a vector.
If $p=2$, then we actually do have $$\|u\|_{W^{k,2}}^2=\|u\|_{L^2}^2+\|Du\|_{L^2}^2+\|D^2u\|_{L^2}^2+\ldots+\|D^ku\|_{L^2}^2,$$ which resembles closely your first case (where $\|D^ku\|_{L^p}:=\||D^ku|\|_{L^p}$).