Problem: Evaluate:
$$I=\int_0^{\pi/2} \ln(\sin(x))\tan(x)dx$$
I tried to attempt it by using the Beta, Gamma and Digamma Functions. My approach was as follows:
$$$$ Consider $$I(a,b)=\int_0^{\pi/2} \sin^a(x)\sin^b(x)\cos^{-b}(x)dx$$ $$$$ $$=\dfrac{1}{2}\beta\bigg (\dfrac{a+b+1}{2},\dfrac{1-b}{2}\bigg )= \dfrac{1}{2} \dfrac{\Gamma(\frac{a+b+1}{2})\Gamma(\frac{1-b}{2})}{\Gamma(\frac{a+2}{2})}$$ Now, $$\dfrac{1}{2}\dfrac{\partial}{\partial a} \beta\bigg (\dfrac{a+b+1}{2},\dfrac{1-b}{2}\bigg )\bigg |_{a=0,b=1} = \int_0^{\pi/2}\ln(\sin(x))\sin^a(x)\tan^b(x)dx \bigg |_{a=0,b=1} = I$$ Now, $$\dfrac{\partial}{\partial a} \beta\bigg (\dfrac{a+b+1}{2},\dfrac{1-b}{2}\bigg )$$ $$$$ $$=\dfrac{1}{2} \dfrac{\Gamma(\frac{1-b}{2})}{(\Gamma(\frac{a+2}{2}))^2}\bigg (\Gamma '\bigg (\frac{a+b+1}{2}\bigg )\Gamma \bigg ( \frac{a+2}{2}\bigg ) - \Gamma '\bigg ( \frac{a+2}{2}\bigg )\Gamma \bigg (\frac{a+b+1}{2}\bigg )\bigg ) $$ $$$$ $$= \dfrac{1}{2}\dfrac{\Gamma(\frac{1-b}{2})}{\Gamma(\frac{a+2}{2})}\bigg (\psi\bigg ( \frac{a+b+1}{2}\bigg )\Gamma \bigg (\frac{a+b+1}{2}\bigg ) - \psi\bigg ( \frac{a+2}{2}\bigg )\Gamma \bigg (\frac{a+b+1}{2}\bigg )\bigg )$$ $$$$ $$=\dfrac{1}{2} \dfrac{\Gamma(\frac{a+b+1}{2})\Gamma(\frac{1-b}{2})}{\Gamma(\frac{a+2}{2})}\bigg (\psi\bigg ( \frac{a+b+1}{2}\bigg )-\psi\bigg ( \frac{a+2}{2}\bigg )\bigg )$$ $$$$ $$\Rightarrow\dfrac{\partial}{\partial a} \beta\bigg (\dfrac{a+b+1}{2},\dfrac{1-b}{2}\bigg )=\dfrac{1}{2} \dfrac{\Gamma(\frac{a+b+1}{2})\Gamma(\frac{1-b}{2})}{\Gamma(\frac{a+2}{2})}\bigg (\psi\bigg ( \frac{a+b+1}{2}\bigg )-\psi\bigg ( \frac{a+2}{2}\bigg )\bigg )$$
$$$$
$$\Longrightarrow \dfrac{1}{2} \dfrac{\partial}{\partial a} \beta \bigg ( \dfrac{a+b+1}{2} ,\dfrac{1-b}{2} \bigg ) \bigg |_{a=0,b=1} = I $$ $$$$ $$=\dfrac{1}{2}\times \dfrac{1}{2} \dfrac{\Gamma(\frac{a+b+1}{2})\Gamma(\frac{1-b}{2})}{\Gamma(\frac{a+2}{2})}\bigg (\psi\bigg ( \frac{a+b+1}{2}\bigg )-\psi\bigg ( \frac{a+2}{2}\bigg )\bigg ) \bigg |_{a=0,b=1}$$ $$$$ $$ =\dfrac{1}{4}\dfrac{\Gamma(0)\Gamma(1)}{\Gamma(1)}\bigg (\psi\bigg ( 1 \bigg )-\psi\bigg (1\bigg )\bigg )$$ $$$$ Could somebody tell me where I have gone wrong?
OK, I realize this does not really answer your question (but look at the update below), in the sense it does not point at your error. Anyways, I think that you should be flexible with your methods doing integrals, so here is how it can be done with the change of variables I suggested:
You end up with the integral $$ I=\int_0^1 \log u\frac{u}{1-u^2}\,du. $$ Now writing $$ \frac{1}{1-u^2}=1+u^2+u^4+\cdots, $$ and using that (integrating by parts) $$ \begin{aligned} \int_0^1 u^{2k+1}\log u\,du&=\bigl[\frac{u^{2k+2}}{2k+2}\log u\bigr]_0^1-\frac{1}{2k+2}\int_0^1 u^{2k+2}\frac{1}{u}\,du\\ &=-\frac{1}{4}\frac{1}{(k+1)^2}. \end{aligned} $$ Thus, $$ I=-\frac{1}{4}\sum_{k=0}^{+\infty}\frac{1}{(k+1)^2}=-\frac{\pi^2}{24}. $$ The last equality by the Basel problem.
Updated version
The problem in your calculation, by inserting $a=0$ and $b=1$, you get a $\Gamma(0)$ (which is undefined, or as its best $\pm\infty$ depending on if you let $b\to 1$ from left or right) times something that is zero. This is indeterminate, and you must look at limits. One way is as follows:
If you first let $a=0$, and write $b=1+\epsilon$, then you get $$ \frac{1}{4}\Gamma(-\epsilon/2)\Gamma(1+\epsilon/2)\bigl(\Psi(1+\epsilon/2)-\Psi(1)\bigr) $$ We multiply and divide by $\epsilon/2$ (lethal weapon number 2), to write this as $$ \frac{1}{4}\frac{\epsilon}{2}\Gamma(-\epsilon/2)\Gamma(1+\epsilon/2)\frac{\Psi(1+\epsilon/2)-\Psi(1)}{\epsilon/2} $$ By letting $\epsilon\to 0$, you will find that the limit is $$ \frac{1}{4}\times (-1)\times 1 \times \frac{\pi^2}{6}=-\frac{\pi^2}{24}. $$ Here, we have used the facts that $\lim_{x\to 0} x\Gamma(x)=1$ and $\Psi^{(1)}(1)=\pi^2/6$.