Let $I$ be an ideal of a commutative ring $R$. P. Aluffi says that for all $j,k$ we have that $$I^j I^k\subseteq I^{j+k}.$$ Since $I^k$ is defined as being $I\cdot I\cdot\dotsc\cdot I$ ($k$ times), I don't see why we wouldn't have equality. What am I missing here?
Also, anything changes if $R$ is not commutative?
You aren't missing anything. It is true that $I^j I^k = I^{j+k}$ (and it follows immediately from the fact that the ideals of $R$ form a monoid with respect to multiplication). The reason why Aluffi was only writing the $\subseteq$ sign is probably that he was only needing the $\subseteq$ sign and thus avoiding a little mental detour. (If you give me the exact location of the statement, I can comment with greater certainty.)
And yes, this all remains true when $R$ is noncommutative. Indeed, if $U$ and $V$ are two ideals of $R$, then the $\mathbb{Z}$-submodule of $R$ spanned by all products of the form $uv$ with $u \in U$ and $v \in V$ is again an ideal of $R$, and therefore the product operation is well-defined; its associativity can be easily seen as a consequence of the associativity of multiplication of $R$.
Furthermore, this all remains true when $R$ is nonunital, as long as $j$ and $k$ are positive. (We could still define $I^0$ to be $R$, but it would not be a very useful thing to do, since $R$ is no longer the neutral element of the semigroup of ideals of $R$ when $R$ is nonunital.)