I would like to consider two ways to compute the (real) integral $\int_{0}^{1}x^{\frac{1}{3}}(1-x)^{\frac{2}{3}}dx$ using complex analysis:
(i) By residues
(ii) By Beta function
My computations:
(i) First of all, my intuition is that the value of the integral should be either $0$ or another real value.
$x^{\frac{1}{3}}(1-x)^{\frac{2}{3}}$ has two singularities in 0 and 1. Therefore, $\sqrt[3]{z(1-z)^2}$ is well defined on $\mathbb{C}\setminus [0,1]$ and $\int_{|z|=\varepsilon}\sqrt[3]{z(1-z)^2}dz \leq 2\pi \varepsilon k \rightarrow 0$, as $\varepsilon \rightarrow 0$, for some constant $k$.
$\int_{\gamma_\varepsilon}\sqrt[3]{z(1-z)^2}dz\rightarrow 3\int_0^1 x(1-x)^2dx$, as $\varepsilon \rightarrow 0$.
Now we need to find the Laurent expansions at the singularities.
For $z=0$
$\sqrt[3]{z(1-z)^2} = \sqrt[3]{z^3-2z^2+z}=z\sqrt[3]{1-\frac{2}{z}+\frac{1}{z^2}}$ and fixing $w=\frac{2}{z}-\frac{1}{z^2}$. Hence, using the binomial formula for $(1-w)^{1/3}$, we get
$(1-w)^{1/3}=1+\frac{w}{3}-\frac{w^2}{9}+\frac{5w^3}{3}+\dots$
Plugging back the value of $w$:
$z(1-w)^{1/3}=z(1+\frac{\frac{1}{z}(2-\frac{1}{z})}{3}-\frac{\frac{1}{z^2}(2-\frac{1}{z})^2}{9}+\dots)=z+\frac{2-1/z}{3}-\frac{(2-1/z)^2}{9z}+\dots$
As a result, the residue is $(-\frac{1}{3}-\frac{4}{9})\frac{1}{z}=-\frac{7}{9}\frac{1}{z}$
For $z=1$
$\sqrt[3]{z(1-z)^2}=\sqrt[3]{(z-1)^2(z-1+1)}=\sqrt[3]{(z-1)^3+(z-1)^2}=(z-1)\sqrt[3]{1-\frac{-1}{z-1}}=(z-1)(1-\frac{1}{3(z-1)}-\frac{1}{9(z-1)^2}+\dots)$
Hence, the residue for $z=1$ is $-\frac{1}{9}$
By the Cauchy-Riemann integral formula, we have
$\int_{|z|=\varepsilon}\sqrt[3]{z(1-z)^2}dz=2\pi i(-\frac{8}{9})$ which is complex-valued.
(ii) Using the Beta function $f(m,n)=\int_0^1x^{m-1}(1-x)^{n-1}=\frac{\Gamma(m)\Gamma(n)}{\Gamma(m+n)}$, for $m-1=1/3$ and $n-1=2/3$, we get:
$\int_0^1 x^{\frac{1}{3}}(1-x)^{\frac{2}{3}}dx=\frac{\Gamma(4/3)\Gamma(5/3)}{\Gamma(3)}=\frac{\frac{1}{3}\Gamma(\frac{1}{3})\frac{2}{3}\Gamma(\frac{2}{3})}{2!}=\frac{1}{9}\Gamma(\frac{1}{3})\Gamma(1-\frac{1}{3})=\frac{1}{9}\frac{\pi}{\sin\frac{\pi}{3}}=\frac{1}{9}\pi\frac{2}{\sqrt{3}}=\frac{2\sqrt{3}\pi}{27}$
Could someone please help me to understand where the mistake is?
You seem to want to use Cauchy's residue theorem (not to be confused with the Cauchy-Riemann equations, BTW) to evaluate this integral. To use the residue theorem, you need to integrate over a closed contour in the complex plane. As far as I can tell, you don't have one; you're just adding up all the residues that you can find. This is not a valid approach.
It is probably possible to find a closed contour in the complex plane that allows you to evaluate this integral. Note, however, that the integrands has branch points at 0 and 1, which means that you have to be careful in what counts as a "closed contour" for the purposes of the residue theorem.