More specifically, $R$ is a commutative ring.
I'm trying to understand what the ideal "generated by $a$" is, where $a$ is an element of $R$. I believe this ideal is simply the set $\{a\cdot u\mid u\in R \}$
(I'd like to say the ideal is just all "multiples" of $a$ but am not sure whether that'd be correct).
Can anyone confirm or deny that what I'm thinking is right?
Yes, that is correct, assuming your ring is commutative with unit. As a consequence, $aR$ is one of the common notations for the ideal generated by $a$, alongside $\langle a\rangle$ and $(a)$. "All multiples of $a$" isn't really wrong to say either, but no one says it because "the ideal generated by $a$" has become conventional and universal.
If $R$ doesn't have a unit, then $aR = \{a\cdot r\mid r\in R\}$ doesn't necessarily contain $a$, so it is not the ideal you're looking for. For instance, in the ring $R = 2\Bbb Z$ of even integers, "all multiples of $2$" will be the ring $4\Bbb Z$, which doesn't contain $2$.