We want to prove that if $R$ is an integral domain (with identity element $1_R$), then $a$ is a proper factor of $b$ (a proper factor meaning, there exists $c$ in $R$ such that $b = ac$, and $c$ is not a unit in $R$) if and only if $Rb \subsetneq Ra$ is true.
To prove this we have to prove both directions, I know how to prove the forward direction but having problems with the backwards direction.
A weaker condition can be proved by the following argument:
Since $Rb$ is a subset of $Ra$, if take some identity element $1_R$, it is trivial to see that $1_Rb = ka$, for some element $k$ in $R$, hence in this case it's obvious $b$ is a factor of $a$. However this only proves a certain special case of this statement, and the condition is also weaker.
I'm wondering if anyone have an idea how to start proving this fact?
Starting where you left off, the only thing remaining is to show $k$ isn't a unit.
If to the contrary it were a unit, then $bk^{-1}=a$ Implies $a\in bR$, hence $aR\subseteq bR$, and equality, contradicting the assumption the containment is proper.
Thus, you have the property in your second line.