Below is factoring a polynomial using method of undetermined coefficient.
While solving the system, my textbook (not English) says it is OK to think that $a>0, k>0$. Why? Why cannot it be like that: $a<0, k<0$ ? Ok, they both shall have same sign, but why do we consider only "both positive"?
Is there any reasoning that allows us skip considering the other possiblility ($a<0, k<0$) ?

Because the author of the book wants integer coefficients. The second case it's $a=k=-1.$
There is much more stronger method:
For all real $k$ we have: $$2x^4-x^3-9x^2-x+1=\left(kx^2-\frac{1}{2}x+1\right)^2-x^2\left((k^2-2)x^2-(k-1)x+\left(2k+\frac{37}{4}\right)\right).$$ Now, we ca choose a value of $k$ such that $k^2-2>0$ and $$(k^2-2)x^2-(k-1)x+\left(2k+\frac{37}{4}\right)=(ax+b)^2$$ for some reals $a$ and $b$.
For which we need also $$(k-1)^2-4(k^2-2)\left(2k+\frac{37}{4}\right)=0,$$ which gives $k=-\frac{3}{2}.$
Id est, $$2x^4-x^3-9x^2-x+1=\left(-\frac{3}{2}x^2-\frac{1}{2}x+1\right)^2-x^2\left(\frac{1}{4}x^2+\frac{5}{2}x+\frac{25}{4}\right)=$$ $$=\left(\frac{3}{2}x^2+\frac{1}{2}x-1\right)^2-\left(x\left(\frac{1}{2}x+\frac{5}{2}\right)\right)^2=(x^2-2x-1)(2x^2+3x-1).$$