I want to find the annihilator of $a'$ and $b'$ of the quotient ring $R=\mathbb{Z}/(a'b')$ where $a',\,b'>1$.
So if I go by the definition, $ann(a')=\{r\in\mathbb{R}\mid a'r=0\}=\{a'\mathbb{Z}+b'\mathbb{Z}+(a'\,b')\in\mathbb{R}\mid a'(a'\mathbb{Z}+b'\mathbb{Z})=0\}$
Am I correct unto this point?
No, there are a couple of problems.
First of all, the $r\in \Bbb R$ part is not right (maybe just a typo?) It should read
$ann(a')=\{x\in\mathbb{Z}/(a'b')\mid (a'+(a'b'))x=0\}$
Secondly, it looks like you're misunderstanding the last part. If I were to rewrite the whole line, I would write this:
$$ann(a')=\{x\in\mathbb{Z}/(a'b')\mid (a'+(a'b'))x=0\}=\{z+(a'\,b')\mid (z+(a'\,b'))(a'+(a'b'))=0+(a'b')\}$$
Rewriting the expression on the far right again, we're trying to determine when $(z+(a'\,b'))(a'+(a'b'))=za'+(a'b')=0+(a'b')$. The last equality happens iff $za'\in (a'b')$.
Then $za'\in (a'b')$ implies $za'=a'b'c$ for some $c\in \Bbb Z$. Cancelling $a'$ from both sides, we arrive at $z=b'c$. Thus $z\in (b')$. So anything that annihilates $a'+(a'b')$ must lie in $(b')+(a'b')$.
Conversely, it's obvious that anything in $(b')+(a'b')$ annihilates $a'+(a'b')$.