This question is about the solution of exercise 1.20 in Elman, Silvester, Wathen. Finite Elements and Fast Iterative Solvers. (The first Chapter of the book is open access and available, for example, here).
Statement: Given that $u \in \mathcal{H}^s(\Omega) \Leftrightarrow \| D^su \| < \infty$, show that the function $u(r, \theta)=r^{2/3}\sin((2\theta+\pi)/3)$ defined on the pie-shaped domain $\Omega$ where $0 \leq r \leq 1$ and $-\pi \leq \theta \leq \pi$ is in $\mathcal{H}^1(\Omega)$ but not in $\mathcal{H}^2(\Omega)$.
Notation: In the book, $D^1u$ and $D^2u$ are defined as the sum of the squares of the first and second derivatives of $u$. For a two-dimensional domain they are given explicitly:
$$ \|D^1 u\|^2 = \int_{\Omega} \left( \left(\frac{\partial u}{\partial x}\right)^2 + \left(\frac{\partial u}{\partial y}\right)^2 \right) $$
$$ \|D^2 u\|^2 = \int_{\Omega} \left( \left(\frac{\partial^2 u}{\partial x^2}\right)^2 + \left(\frac{\partial^2 u}{\partial x \partial y}\right)^2 + \left(\frac{\partial^2 u}{\partial y^2}\right)^2 \right) $$
My (wrong) solution: Let us compute the $L^2$ norm of $D^1u$ and $D^2u$ using polar coordinates:
$$ \|D^1 u \|^2 = \frac{4r^{1/3}}{9}\int_0^1\int_{-\pi/2}^{\pi} \left( r^{2} \cos\left( \frac{(\pi +2 \theta)}{3} \right)^2 + \sin\left( \frac{(\pi +2 \theta )}{3} \right)^2 \right) r\,dr\,d\theta = \frac{7 \pi}{20} $$
$$ \|D^2 u \|^2 = \frac{4}{81r^{5/3}}\int_0^1\int_{-\pi/2}^{\pi} \left( 4 r^2 \cos\left(\frac{(\pi +2 \theta)}{3}\right)^2 + (1 + 4r^4)\sin\left(\frac{(\pi +2 \theta )}{3}\right)^2 \right) r\,dr\,d\theta = -\frac{\pi}{90} $$
At this point, I'm a bit puzzled by the negative squared norm. However, I don't see where my mistake is, and any help would be much appreciated.
Problems
Since $r$ is a variable of integration, moving it outside of the integral (as you did with $r^{1/3}$ and $r^{5/3}$ is a wrong move.
Also, the derivatives in rectangular coordinates, such as $\dfrac{\partial ^2u }{\partial x\partial y}$, do not become $\frac{\partial ^2u }{\partial r\partial \theta}$ when expressed in polar coordinates. The multivariable chain rule must be used to move between coordinate systems, and the result can be complicated.
Finally, it seems that you did not square the mixed partial derivative in your second integral.
Solution
Instead of computations, notice that $u$ is homogeneous: $$u(tx,ty)=t^{2/3}u(x,y), \quad t>0$$ From the chain rule it follows that $$\|Du\|(tx,ty)=t^{-1/3}\|Du\|(x,y), \qquad \|D^2u\|(tx,ty)= t^{-4/3}\|D^2u\|(x,y) $$
When integrating in polar coordinates, integrate over $\theta$ first because the result will be some positive number times a power of $r$. Specifically,
$$\int_0^1 \int_{-\pi}^{\pi} \|Du\|^2 \,d\theta\,r\,dr =\int_0^1 C_1r^{-2/3}r\,dr$$ and $$\int_0^1 \int_{-\pi}^{\pi} \|D^2u\|^2 \,d\theta\,r\,dr =\int_0^1 C_2r^{-8/3}r\,dr$$ The first integral converges, the second doesn't.