I found an algorithm book which tries to define asymptotic notations as sets and then used notations like $n=O(n^2)$. Is there a mathematically correct way to define asymptotic notations like $O(n), \Omega(n), \Theta(n), o(n)$ and $\omega(n)$? And is there an algorithm books that uses notations correctly?
I found something about http://www.artofproblemsolving.com/Forum/viewtopic.php?f=296&t=31517&start=20 but the problem is that I don't know rigorous definitions of class and set and difference of those so I can't say if it is a correct approach to formalize the $O$ notation.
You should probably check this wiki page
$n=O(n^2)$ is mathematically correct because $\frac{n}{n^2}=\frac{1}{n} < 1$. It is also true that $n=o(n^2)$ as $\lim_{n\rightarrow +\infty} |\frac{n}{n^2}|=0$
I don't have any reference on the subject, but the books I've seen usually use the notations properly, even if they don't go into much details about what the symbol really means. You just need to remember how to compare powers, ln and exp and you'll be fine 99% of the time.