I'm trying to evaluate the limit $\displaystyle\lim_{x\to\infty}x-\sqrt{(x-a)(x-b)}$
Since there is a square root of a product namely $\sqrt{(x-a)(x-b)}$ I think it's useful to use $\text{GM}\le\text{AM}$ for $(x-a)$ & $(x-b)$: $\sqrt{(x-a)(x-b)}\le\dfrac{(x-a)+(x-b)}{2}=x-\dfrac{a+b}{2}\tag*{}$
To see whether the function $f(x)=x-\sqrt{(x-a)(x-b)}$ is increasing or decreasing,
\begin{align}f'(x)&=1-\dfrac{2x-(a+b)}{2\sqrt{(x-a)(x-b)}}\\&=1-\dfrac{\dfrac{x-(a+b)}2}{\sqrt{(x-a)(x-b)}}\\&=1-\dfrac{\text{AM}}{GM}\\&\le0 \because\text{ GM}\le\text{AM}\end{align}
hence $f(x)=x-\sqrt{(x-a)(x-b)}$ is decreasing. Then in order for the existence of the limit, $f(x)$ must be bounded below.
Here I again use the same inequality:
\begin{align}\\&\sqrt{(x-a)(x-b)}\le\dfrac{(x-a)+(x-b)}{2}=x-\dfrac{a+b}{2}\\&\implies f(x)=x-\sqrt{(x-a)(x-b)}\ge\dfrac{a+b}{2}\end{align}
If I'm right, then the limit exists and $\displaystyle\lim_{x\to\infty}x-\sqrt{(x-a)(x-b)}=\dfrac{a+b}2$
Another query is that in the inequality $\sqrt{(x-a)(x-b)}\le\dfrac{(x-a)+(x-b)}{2}$ the equality holds iff $x-a=x-b$- how to solve it for $x$? I want to know where $f(x)$ takes its minimum.
You are correct, but the limit is easier to evaluate if you consider $$x-\sqrt{(x-a)(x-b)}=\frac{x^2-(x-a)(x-b)}{x+\sqrt{(x-a)(x-b)}}= \frac{(a+b)-ab/x}{1+\sqrt{(1-a/x)(1-b/x)}}$$ for $x> \max(a,b,0)$.