Let $R$ be a commutative ring. Then an $R$-algebra $R$ is a ring $A$ such that $A$ is an $R$-module (write it as left module) and such that the multiplication in the ring $A$ is compatible with the module structure, i.e.
$$\forall a ,b \in A, \forall r \in R: (ra)b = r(ab) = a(rb)$$
First question: Is this definition of algebra correct?
I was playing with this definition and observed the following: we can evaluate an expression $(ra)(sb)$ with $r,s \in R, a,b \in A$ in two ways:
$$(ra)(sb) = r(a(sb)) = r(s(ab)) = (rs)(ab)$$ $$(ra)(sb) = s((ra)b) = s(r(ab)) = (sr)(ab)$$
Second question: Is the following calculation above correct? If yes, is this the reason we ask that $R$ is a commutative ring?
Your definition corresponds to what are called "associative algebras". Depending on the domain of math one works in, algebras may be implicitly understood to be always associative (and possibly also with unit), but in other areas this might not be the case (for instance if you are interested in Lie algebras, or octonions).
Your second computation is correct and indeed this is a simple way to understand why we want the base ring to be commutative. As far as I know, there is no commonly accepted notion of an associative algebra over a non-commutative ring, and this kind of computation shows why it's very awkward to define.
Basically, you want your scalars in $R$ to commute with elements in $A$, but that won't make much sense unless they already commute with each other.