Here is the function which could easily be solved using expansion method but how could I solve it using integration by parts
$$\int y^4(1-y)^3 dy$$
The problem is, when I apply integration by parts to solve it, it is never ending solution and I am not able to get the answer.
For example, I let $u = (1-y)^3$ and $dv = (y^4)$, so $du = 3(1-y)^2$ and $v = \dfrac{y^5}{5}$
When I apply the Integration by Parts formula,
$$uv - \int v du$$
I got the kind of same equation as I started with, so I need to apply integration by parts once again, and then again. How many times is it required to apply before I get the answer ?
$$I(4,3) = \int y^4 (1-y)^3 \mathrm{d}y$$
You were heading in the right direction, i.e. $\mathrm{d}v=y^4 \Rightarrow v = \frac{y^5}{5}$
$$ \begin{align*} I(4,3) &= \frac{y^5(1-y)^3}{5} + \frac{3}{5} \int y^5 (1-y)^2 \mathrm{d}y\\ &= \frac{y^5(1-y)^3}{5} + \frac{3}{5} I(5,2)\\ \end{align*} $$
Similarly use $\mathrm{d}v=y^5$, $u=(1-y)^2$ to evaluate $I(5,2)$
$$ \begin{align*} I(5,2) &= \frac{y^6(1-y)^2}{6} +\frac{1}{3} I(6,1)\\ &= \frac{y^6(1-y)^2}{6} + \frac{1}{3}\left(\frac{y^7}{7} - \frac{y^8}{8}\right)\\ I(4,3) &= \frac{y^5(1-y)^3}{5} + \frac{1}{10}y^6(1-y)^2+\frac{1}{5}\left(\frac{y^7}{7} - \frac{y^8}{8}\right)+ Constant \end{align*} $$