I studied mathematics about two decades ago, but unfortunately, I remember little of it. I'm hoping to start studying cryptography and signal processing, but I'm not entirely sure what course/self-study sequence to follow. My initial ideas are something along these lines:
- Number theory
- Linear algebra and multivariable calculus
- Abstract algebra and probably some coding theory
- Analysis, both real and complex
- ...
Given how little I remember, I'm not sure how much overlap I might be missing. For example, I know that coding theory utilises quite a bit of group theory, but I don't quite know the extent of the overlap.
For cryptography, you need some math courses in number theory and abstract algebra, but it sounds like you know about those.
We have three main cartographic ciphers in the form of symmetric ciphers (algorithmic based like the Advanced Encryption Standard - AES), asymmetric ciphers and stream ciphers.
Asymmetric ciphers are based on 'trap-door' mathematical functions that use the integer factorization problem (like RSA), discrete log problem (like Diffie-Hellman) or elliptic curve discrete log problem (elliptic curve crypto).
Cryptography Books
A Course in Number Theory and Cryptography, Neal Koblitz
Algebraic Aspects of Cryptography, Neal Koblitz
Elliptic Curves: Number Theory and Cryptography, Second Edition (Discrete Mathematics and Its Applications), Lawrence C. Washington
The mathematics of ciphers: number theory and RSA cryptography, Coutinho S C, Severino Collier Coutinho
An Introduction to Cryptography, Richard A. Mollin
Introduction to Cryptography with Coding Theory (2nd Edition) by Wade Trappe and Lawrence C. Washington (Jul 25, 2005)
Handbook of Applied Crytography, Alfred Menezes, Paul van Oorschot, Scott Vanstone
Guide to Elliptic Curve Cryptography, Darrel Hankerson, Alfred Menezes, and Scott Vanstone
Practical Cryptography, Niels Ferguson, Bruce Schneier
Also note that there is also the field of cryptanalysis, which is more statistically and probability theory focused.
You might want to peruse the MIT Open Courseware on Crypto for further guidance on courses and areas to brush up on.
Here are some notes and here is another set of notes (and you can find other very good mathematically based ones online too).
Here is a list of topical areas that involve crypto to get an idea of scope.
For cryptanalysis read these from an expert.
That should be enough to help guide you forward and to help you figure out what courses you need.
Regards