In Serre's A Course In Arithmetic, it says the following:
I don't know what it is talking about, I know the definition of $f'$, but what is This is in the last sentence refered to? $f'$ is a function on $V$, not $V'$. So what is such rapid decreasing function $g$ that have domain on $V'$?
BTW, it would be helpful if someone can also explained such measure $\mu$ is $invariant$ by what function?
This definition is a little fast-paced and can be easily misunderstood. Here, $y\in V'$ so that $f'$ is indeed defined on $V'$. The way you can see this is the dual pairing in the exponent of $e$: $\langle x,y\rangle$ implicitly tells you that $y$ is in the dual space where $x$ comes from. Hence the "this is" that you refer to is actually $f'$ as defined on $V'$.
As for what invariance means, it is likely that it means invariant under translations, i.e. $\mu(E+x) = \mu(E)$ for any Borel set $E$ in $V$ and $x\in V$. This is key to the really nice algebraic structure of the Fourier transform.