Suppose there is a sequence
$$u_n = au_{n-1} - a^2u_{n-1}^2 + bu_{n-2} - b^2u_{n-2}^2$$
with the boundary condition, $u_0, u_1$ both are positive and less than $1$.
How can I show that this sequence is convergent?
Actually, my goal is to show that $u_n$ converges to a value greater than $0$.
Possibly, when $a + b > 1$ then $\lim\limits_{n\rightarrow \infty} u_n \neq 0$ as I experiment with a computer progam.
This is a long comment, or possibly half an answer.
If the sequence has a limit $L$ then $$L=aL-a^2L^2+bL-b^2L^2\tag{1},$$which can be rearranged viz. $0=L(a+b-1-(a^2+b^2)L)$. Therefore $L$ is nonexistent, $L=0$ or $$L=\frac{a+b-1}{a^2+b^2}\tag{2},$$the latter being the only case in which $u_n$ converges to a non-zero value. Indeed, in this case that value is positive iff $a+b>1$, as you've conjectured.
The hard part is checking whether the sequence has a limit. Define $$\epsilon_n:=u_n-L=a(\epsilon_{n-1}+L)-a^2(\epsilon_{n-1}+L)^2+b(\epsilon_{n-2}+L)-b^2(\epsilon_{n-2}+L)^2-L.$$This simplifies for a root of (1) to $$\epsilon_n=a(1-2aL)\epsilon_{n-1}-a^2\epsilon_{n-1}^2+b(1-2bL)\epsilon_{n-2}-b^2\epsilon_{n-2}^2\tag{3}.$$The real question is whether (2) and (3) imply $\epsilon_n\to 0$. I suspect that can be proved from your boundary conditions (possibly with some further requirement on $a,\,b$), but I'm not sure how. But if any proof strategy works, it'll be that one.