I took out x from the square root and reached the following expression, $$\lim_{x\to -\infty} x\sqrt{1-\frac{1}{x}+\frac{1}{x^2}} + ax - b = 0$$ then I separated the part of the expression which contains x from b and tried evaluating its limit, giving me the following expression, $$\lim_{x\to -\infty} \frac{\sqrt{1-\frac{1}{x}+\frac{1}{x^2}}+a}{\frac{1}{x}} $$ Now, the problem is that I am unable to proceed from here. What I have read and learnt till now tells me that since the denominator of this expression tends to 0 the numerator must also tend to 0 for the limit to exist as a finite number, but I don't understand why only 0? Can't it be something else? Can someone please explain me the approach to solving such questions and the reason behind why it works? And it would be great if you could tell me the values of a and b in this case.
Note: Assuming that the numerator must also tend to 0 for the limit to be finite, and then applying L'Hopital's rule I got the values of a and b as -1 and -1/2 respectively. The only thing is that I don't have the answer of this question with me, so it would be great if you could tell me what I am getting is right or wrong.
Let me address your question on how or why it works that way.
If the numerator were to converge to $\pm\infty$, then the absolute value of the numerator increases without bound, and this happens while the denominator gets smaller and smaller, so we expect the quotient of the two to get really massive as $n \to \infty$ without bound. Hence, in this case, the limit converges to $\pm\infty$.
If the numerator were to converge to some non-zero finite number, then the limit converges to some $\frac{c}{0}$ for some finite number $c$. That's essentially infinite as well. In rigorous terms, the limit converges also to $\pm\infty$ depending on the sign of $c$.
Hence, for the limit to exist, the numerator indeed needs to converge to zero, so at least it can catch up with how the denominator converges to $0$. With this, the limit has a higher chance of converging to a finite number. In this case, the limit converges to the indeterminate form $\frac{0}{0}$ which calls for the L'Hospital rule to evaluate the limit.
Update
To address OP's follow-up comment, let's get back to the question. We want to find out for which values of $a$ and $b$ does the limit equation
$$\lim_{x \to -\infty} \sqrt{x^2 - x + 1} + ax - b = 0$$
hold. In this answer, I'll try to follow OP's apparent strategy in solving the equation.
Indeed, this limit equation holds if and only if
$$\lim_{x \to -\infty} \sqrt{x^2 - x + 1} + ax = b$$
This is the beginning of OP's attempt to solve the equation. The question reduces to finding the values of $a$ and $b$ for which the limit
$$\lim_{x \to -\infty} \sqrt{x^2 - x + 1} + ax$$
exists. What OP tried to do is factor out $-x$ to get the equivalent limit
$$\lim_{x \to -\infty} -x \left(\sqrt{1 - \frac{1}{x} + \frac{1}{x^2}} - a\right)$$
Substituting $-x$ into $x$ so that $x \to \infty$, we have
$$\lim_{x \to \infty} x \left(\sqrt{1 + \frac{1}{x} + \frac{1}{x^2}} - a\right)$$
Here, multiplying by $x$ is equivalent to dividing by $1/x$, so we have
$$\lim_{x \to \infty} \frac{\sqrt{1 + \frac{1}{x} + \frac{1}{x^2}} - a}{\frac{1}{x}}$$
Clearly, the denominator converges to $0$. So for this limit to exist, the numerator must converge to $0$. Hence, we must have
$$\lim_{x \to \infty} \sqrt{1 + \frac{1}{x} + \frac{1}{x^2}} - a = 0$$
The limit on the left-hand side evaluates easily to $1 - a$, so we have
$$1 - a = 0 \Rightarrow a = 1$$
Now that we got $a$, we can plug it back into the limit equation to solve for $b$ and we can proceed from here.
Update
Let's finish this. Plugging in $a = 1$ into the equation, we have
$$b = \lim_{x \to \infty} \frac{\sqrt{1 + \frac{1}{x} + \frac{1}{x^2}} - a}{\frac{1}{x}} = \lim_{x \to \infty} \frac{\sqrt{1 + \frac{1}{x} + \frac{1}{x^2}} - 1}{\frac{1}{x}}$$
Take $u = 1/x$ so that $u \to 0$ and we have
$$b = \lim_{u \to 0} \frac{\sqrt{1 + u + u^2} - 1}{u}$$
From here, we get the indeterminate form $0/0$ and L'Hospital's rule kicks in.
$$b = \lim_{u \to 0} \frac{2u + 1}{\sqrt{1 + u + u^2}} = \frac{1}{2}$$
Hence, we have $a = 1$ and $b = 1/2$.