From the fact that $f(x)= \left[f( x) \right]+ \left\{ f(x) \right\} $, where $ \left\{ x \right\} $ is the fractional part function, one can write by a direct substitution for the function $M(x)=\sum_ {n\leq x}\mu(n)$, where $\mu(n)$ is the Möbius function that $$M(x)=\sqrt{x} \left[ \frac{M(x)}{\sqrt{x}} \right]+\sqrt{x} \left\{ \frac{M(x)}{\sqrt{x}} \right\} . $$ On the other hand we know the following
Proposition. For $x>1$, $$\sum_{n\leq x}M \left( \frac{x}{n} \right) =1.$$
Then I can compute $$\sum_{n\leq x}\sqrt{\frac{x}{n}} \left[\sqrt{\frac{n}{x}} M \left(\frac{x}{n} \right) \right] =O \left( \sqrt{x} \right), $$ with the obvious bound for the fractional part function, and since $O \left( \sqrt{x}\sum_{n\leq x} \frac{1}{\sqrt{n}}\right)=O(\sqrt{x}) $.
But I don't know if such an asymptotic has a mathematical sense. Is it possible to improve it much more? I didn't draw a plot of this function.
Question. Is it possible for $x>1$ to compute or to improve: $$\sum_{n\leq x}\sqrt{\frac{x}{n}} \left[\sqrt{\frac{n}{x}} M \left(\frac{x}{n} \right) \right] =O \left( \sqrt{x} \right)\,?$$
It is not true that $$\sqrt{x}\sum_{n\leq x}\frac{1}{\sqrt{n}}\ll\sqrt{x} $$ since $$\sum_{n\leq x}\frac{1}{\sqrt{n}}\geq\frac{1}{\sqrt{x}}\sum_{n\leq x}1=\sqrt{x} $$ and from Abel's summation we have $$\sum_{n\leq x}\frac{1}{\sqrt{n}}=\frac{x}{\sqrt{x}}+\frac{1}{2}\int_{1}^{x}\frac{\left[t\right]}{t^{3/2}}dt\ll\sqrt{x} $$ so your trivial bound is $$ \begin{align}\sum_{n\leq x}\sqrt{\frac{x}{n}}\left[\sqrt{\frac{n}{x}}M\left(\frac{x}{n}\right)\right]= & \sqrt{x}\sum_{n\leq x}\frac{1}{\sqrt{n}}\left(\sqrt{\frac{n}{x}}M\left(\frac{x}{n}\right)-\left\{ \sqrt{\frac{n}{x}}M\left(\frac{x}{n}\right)\right\} \right) \\ = & 1-\sqrt{x}\sum_{n\leq x}\frac{1}{\sqrt{n}}\left\{ \sqrt{\frac{n}{x}}M\left(\frac{x}{n}\right)\right\} \\ \ll & x. \end{align} $$ I think it is not simple to do better. Note that even if we assume the Riemann hypothesis we have, for all $\epsilon>0$ , $$\sqrt{\frac{n}{x}}M\left(\frac{x}{n}\right)\ll\left(\frac{x}{n}\right)^{\epsilon} $$ and also holds $$\liminf_{x\rightarrow\infty}\frac{M\left(x\right)}{\sqrt{x}}<-1.009,\,\limsup_{x\rightarrow\infty}\frac{M\left(x\right)}{\sqrt{x}}>1.06. $$