Find positive $K$ such that $$\int_0^\infty\left(\frac{1}{\sqrt{2x^2+1}}-\frac{K}{x+1}\right)dx$$ converges
I used the fact that $\frac{1}{\sqrt{2x^2+1}}-\frac{K}{x+1}>\frac{-K+1/\sqrt2}{1+x}$ for $x>1$ to prove it diverges for $K < 1/\sqrt2$.
And the fact that $\frac{1}{\sqrt{2x^2+1}}-\frac{K}{x+1}<\frac{1-K}{x+1}$ to prove it diverges for $K>1$.
However I am not sure how to prove it converges (or not) in $[\frac1{\sqrt2},1]$.
I think it would not converge for any $K$ becase the denomiantor is sorta linear and this would never be identically $0$ for any $K$.
Your bounds are correct but too weak. Here we need a more precise asymptotic analysis.
Note that as $x\to +\infty$, $$\begin{align*}\frac{1}{\sqrt{2x^2+1}}-\frac{K}{x+1}&=\frac{x+1-\sqrt{2}Kx\sqrt{1+\frac{1}{2x^2}}}{(x+1)\sqrt{2x^2+1}}\\ &=\frac{x+1-\sqrt{2}Kx(1+\frac{1}{4x^2}+o(\frac{1}{x^2}))}{\sqrt{2}x^2+O(x)}\\ &=\frac{(1-\sqrt{2}K)x+1+O(1/x)}{\sqrt{2}x^2+O(x)}.\end{align*}$$ What may we conclude?