The questions below assume the following definitions of the Mellin transform and associated convolutions.
(1) $\quad F(s)=\mathcal{M}_x[f(x)](s)=\int\limits_0^\infty f(x)\,x^{s-1}dx$
(2) $\quad f(x)*_{\mathcal{M}_1}g(x)=\int\limits_0^\infty f(x)\,g\left(\frac{y}{x}\right)\frac{dx}{x}$
(3) $\quad f(x)*_{\mathcal{M}_2}g(x)=\int\limits_0^\infty f(x)\,g(y\,x)\frac{dx}{x}$
Assume the following function definitions.
(4) $\quad f_1(x)=\sqrt{x}\cos(2\,\pi\,a\,x)\,,\quad a>0$
(5) $\quad f_2(x)=\frac{1}{\sqrt{x}}\cos(2\,\pi\,a\,x)\,,\quad a>0$
(6) $\quad g(x)=\log(x)$
I've used Mathematica to evaluate the Mellin convolution $f_1(x)*_{\mathcal{M}_1}\log(x)$ both via the $MellinConvolve$ function and via the $Integrate$ function which (after some simplifications and assuming $y>0$) provide consistent results as illustrated below.
(7) $\quad f_1(x)*_{\mathcal{M}_1}\log(x)=MellinConvolve\left[f_1(x),\log(x),x,y\right]=\frac{\log\left(64\,\pi^2\,a^2\,y^2\right)+\pi+2\,\gamma}{4\sqrt{a}},\quad a>0$
(8) $\quad f_1(x)*_{\mathcal{M}_1}\log(x)=\int_0^\infty f_1(x)\frac{\log\left(\frac{y}{x}\right)}{x}\,dx=\frac{\log(64)+2\log(\pi\,a\,y)+\pi+2\,\gamma}{4\sqrt{a}}\,,\quad a>0$
I've also evaluated the alternate Mellin convolution of $f_2(x)*_{\mathcal{M}_2}\log(x)$ which (again after some simplifications) provides the following result.
(9) $\quad f_2(x)*_{\mathcal{M}_2}\log(x)=\int_0^\infty f_2(x)\log(y\,x)\,dx=-\frac{\log(64)+2\log\left(\frac{\pi\,a}{y}\right)+\pi+2\,\gamma}{4\sqrt{a}}\,,\quad a>0\land y>0$
I suspect the results illustrated in (7), (8), and (9) above may be incorrect, but haven't yet been able to confirm it.
Question 1: Are the results illustrated in (7), (8), and (9) above correct?
Question 2: If so, what are the derivations of these results? Can these results be proven via integration by parts?
Question 3: If not, what are the correct results for the Mellin convolution $f_1(x)*_{\mathcal{M}_1}\log(x)$ illustrated in (7) and (8) above and the alternate Mellin convolution $f_2(x)*_{\mathcal{M}_2}\log(x)$ illustrated in (9) above?
I've explored validation of the convolutions above via the following relationships, but note that the Mellin transform of $\log(x)$ is undefined.
(10) $\quad\mathcal{M}_x[f(c\,x)](s)=c^{-s}\mathcal{M}_x[f(x)](s)$
(11) $\quad\mathcal{M}_x[f(x)*_{\mathcal{M}_1}g(x)](s)=\mathcal{M}_x[f(x)](s)\,\mathcal{M}_x[g(x)](s)=F(s)\,G(s)$
Assume the simpler function $f_0(x)$ defined in (12) below (analogous to $f_1(x)$ defined in (4) above) and associated Mathematica evaluations of the Mellin convolution and Mellin transform illustrated in (13) and (14) below.
(12) $\quad f_0(x)=\sqrt{x}\cos(x)$
(13) $\quad f_0(x)*_{\mathcal{M}_1}\log(x)=MellinConvolve\left[f_0(x),\log (x),x,y\right]\\$ $\qquad\qquad\qquad\qquad=\frac{1}{2}\sqrt{\frac{\pi}{2}}(2\log(y)+\log(16)+\pi+2\,\gamma)$
(14) $\quad F_0(s)=\mathcal{M}_x[f_0(x)](s)=\frac{1}{2}(2\,s-1)\cos\left(\frac{\pi\,s}{2}+\frac{\pi}{4}\right)\,\Gamma\left(s-\frac{1}{2}\right),\quad-\frac{1}{2}<\Re(s)<\frac{1}{2}$
The result of the Mellin convolution illustrated in (13) above can be rewritten as follows.
(15) $\quad f_0(x)*_{\mathcal{M}_1}\log(x)=\sqrt{\frac{\pi}{2}} \log \left(4 e^{\gamma +\frac{\pi }{2}} y\right)$
Relationship (10) above can be used to evaluate the Mellin transform of (15) above as follows.
(16) $\quad\mathcal{M}_x[f_0(x)*_{\mathcal{M}_1}\log(x)](s)=\sqrt{\frac{\pi}{2}}\left(4\,e^{\gamma+\frac{\pi}{2}}\right)^{-s}\mathcal{M}_y[\log (y)](s)$
The relationship (11) above and the results illustrated in (14) and (16) above imply the following.
(17) $\sqrt{\frac{\pi}{2}}\left(4\,e^{\gamma+\frac{\pi}{2}}\right)^{-s}\mathcal{M}_y[\log (y)](s)=\frac{1}{2}(2\,s-1)\cos\left(\frac{\pi\,s}{2}+\frac{\pi}{4}\right)\,\Gamma\left(s-\frac{1}{2}\right)\,\mathcal{M}_x[\log(x)](s)$
(18) $\sqrt{\frac{\pi}{2}}\left(4\,e^{\gamma+\frac{\pi}{2}}\right)^{-s}=\frac{1}{2}(2\,s-1)\cos\left(\frac{\pi\,s}{2}+\frac{\pi}{4}\right)\,\Gamma\left(s-\frac{1}{2}\right)$
The left and right sides of (18) above are only equal at $s=0$ which implies the Mathematica evaluation of the Mellin convolution in (13) above is incorrect.
You need a bunch of complex analysis.
For $a > 0$ : $$ \int_0^\infty x^{s-1} e^{-ax}dx= \int_0^\infty (y/a)^{s-1} e^{-y}d(y/a) = a^{-s} \int_0^\infty y^{s-1}e^{-y}dy = a^{-s} \Gamma(s)=e^{-s \log a} \Gamma(s)$$
Note $ \int_0^\infty x^{s-1} e^{-ax}dx-e^{-s \log a} \Gamma(s)$ is analytic for $\Re(a) > 0$, thus by analytic continuation, or the isolated zero theorem, $ \int_0^\infty x^{s-1} e^{-ax}dx-e^{-s \log a} \Gamma(s)=0$ stays true for $\Re(a) > 0$.
For $\Re(s) \in (0,1)$, $ \int_0^\infty x^{s-1} e^{-ax}dx$ is continuous in $a, \Re(a) \ge 0, a \ne 0$. Thus for $b > 0$
$$ \int_0^\infty x^{s-1} \cos(bx) dx=b^{-s} \int_0^\infty x^{s-1} \cos(x) dx\\= \lim_{a \to i} b^{-s}\int_0^\infty x^{s-1} \frac{e^{-ax}}{2} dx+\lim_{a \to -i}b^{-s} \int_0^\infty x^{s-1} \frac{e^{-ax}}{2} dx \\ = \lim_{a \to i}b^{-s}\frac{e^{-s \log a} \Gamma(s)}{2}+\lim_{a \to -i}b^{-s}\frac{e^{-s \log a} \Gamma(s)}{2}\\ = b^{-s}\frac{e^{-s i\pi/2} \Gamma(s)+e^{s i\pi/2} \Gamma(s)}{2}=b^{-s}\cos(\pi s/2) \Gamma(s)$$
Finally for $b > 0, \Re(s) \in (0,1)$ $$ \int_0^\infty x^{s-1} \cos(bx)(\log x) dx = \frac{d}{ds}\int_0^\infty x^{s-1} \cos(bx) dx =\frac{d}{ds}b^{-s}\cos( \pi s/2) \Gamma(s)$$