I need help in evaluating the following contour integral:
$$\frac{1}{2\pi i}\int_{c-i\infty}^{c+i\infty} \frac{\zeta(2s)\zeta(s-1)}{\zeta(2s-2)} \frac{x^{s}}{s} ds $$ It looks like a complicated version of Mertens function: $$ \frac{1}{2\pi i}\int_{c-i\infty}^{c+i\infty} \frac1{\zeta(s)} \frac{x^{s}}{s} \, ds = M(x) $$ but I don't have the skills to solve it...
The Dirichlet series $$ \frac{\zeta(2s)\zeta(s-1)}{\zeta(2s-2)}$$ isn't a complicated version of $\zeta(s)^{-1}$, and has arithmetic meaning. The coefficients of the Dirichlet series include primes whose powers have odd parity.
Your contour integral is an inverse Mellin Transform, a form of Laplace/Fourier transform. This question is really asking to use Perron's Formula to understand the Dirichlet series above.
It is relatively simple to get a poor estimate, and extremely hard (to be read, impossible at the moment) to get a good estimate. But let's see what we can say. We consider
$$\frac{1}{2\pi i}\int_{(c)} \frac{\zeta(2s)\zeta(s-1)}{\zeta(2s-2)} \frac{X^{s}}{s} ds,$$ where initially $c > 10$ is in the region of absolute convergence of the Dirichlet series. The first question one needs to ask is: where are the poles of this object?
The numerator has poles at $s = 2$ and $s = \frac{1}{2}$. The denominator contributes poles at $s = 0$ (which we understand) and at $2s - 2 = \rho$, where $\rho$ is a zero of the zeta function. Rewriting, these are at $s = 1 + \frac{\rho}{2}$. With current understanding of the zeroes of the zeta function, we only know that these additional poles have real part less than $\frac{3}{2}$. If we assumed the Riemann Hypothesis, then these poles have real part $\frac{5}{4}$.
There is no hope of moving the line of integration past the poles coming from zeroes of the zeta function, as there are simply too many and we don't have very much decay.
Let's shift the line of integration to $c = \frac{3}{2} + \delta$ for a $\delta$ to be specified later. We pick up a pole at $s = 2$ with residue $$\operatorname{Res}_{s = 2} \frac{\zeta(2s)\zeta(s-1)}{\zeta(2s-2)} \frac{x^{s}}{s} = \frac{\zeta(4)}{\zeta(2)} \frac{X^2}{2}.$$
As we pass no additional poles, we now ask about the convergence of the integral at $\sigma = \frac{3}{2} + \delta$.
Altogether, initial estimates indicate that the shifted integral looks like $$ X^{\frac{1}{2} + \delta} \lim_{T \to \infty} \int_{\sigma - iT}^{\sigma + iT} \lvert t \rvert^{-\frac{3}{4} - \frac{\delta}{2}} dt.$$ This converges when $\delta > \frac{1}{2}$... which corresponds to us not moving the line of integration past $2$ at all. That's too bad.
Getting anything better requires much more work. But I can tell you what should be possible, if one were to put a lot of work in.
Assuming the Lindelof Hypothesis puts the integral right on the edge of absolute convergence. Heuristically, any amount of oscillation should make the integral converge. There is exactly one source of oscillation, which is the $X^{it}$ part that is lost when we take absolute values. It is extremely likely that this oscillation, if measured and approximated correctly, actually makes the integral converge. [In fact, it's almost certain].
This would lead to the following estimate. Denote your Dirichlet series by $$ \frac{\zeta(2s)\zeta(s-1)}{\zeta(2s - 2)} = \sum_{n \geq 1} \frac{a(n)}{n^s}.$$ Then this would show that $$ \sum_{n \leq X} a(n) = \frac{\zeta(4)}{2\zeta(2)} X^2 + O(X^{3/2}).$$
I should also mention that one can use Mellin Transforms with stronger convergence to give slightly weaker results, but which are more tenable. For instance, without doing any handwaving, oen can use the related Mellin Transform $$\frac{1}{2\pi i}\int_{(c)} \frac{\zeta(2s)\zeta(s-1)}{\zeta(2s-2)} \frac{X^{s}}{s(s+1)} ds$$ and the same back-of-the-envelope calculations as above to show that $$ \sum_{n \leq X} a(n)(1 - \frac{n}{X}) = \frac{\zeta(4)}{6\zeta(2)} X^2 + O(X^{3/2}).$$ This transform and resulting weight is sometimes called a Césaro weighted transform. Although I couldn't find a source to link you to, I know that these appear in Murty's Problems for Analytic Number Theory, as well as very many classical analytic number theory papers.