How "deep" is the theory of encryption keys? Can a "generalist" approach it or does one need to be a number theorist?

Question

How "deep" is the theory of encryption keys? Can a "generalist" approach designing new keys or understanding state of the art "security" or does one need to be a number theorist?

Answer 1

I think there is a bit of confusion here. There is a difference between a cryptosystem, an encryption scheme, and an encryption (or decryption) key.

Nonetheless, the short answer is No, you don't have to be a mathematician or computer scientist to play with the design of cryptosystems. However, you do need to be a mathematician or computer scientist for anyone else to take you seriously if you want to share your work. Also, most cryptographers are not number theorists.

Now a longer answer.

A whole cryptosystem will include an encryption scheme, a decryption scheme, and some method to generate the necessary keys. One might use the encryption scheme to encrypt a message $M$ into some sort of ciphertext $C$, and then use the decryption scheme to decrypt the ciphertext $C$ back into $M$ --- using the keys where necessary.

A good cryptosystem will have easy-to-compute encryption and decryption steps, but have them be hard (i.e. computationally infeasible) to break without the key. Current cryptosystems generally rely heavily on a single "hard problem", something which people reasonably believe is easy-to-compute and hard-to-break. The problems in use today are factoring (or closely related, the RSA problem), computing discrete logarithms, finding shortest vectors in a lattice, finding nearest vectors in a lattice, or (closely related to the previous two, finding a "good basis" for a lattice when given a "bad basis").

Almost every cryptosystem in use relies on one of these hard problems.

It is rather easy to think of new encryption and decryption schemes. If one can show that breaking a proposed scheme reduces to solving one of these hard problems, then people would believe the scheme --- then it's a question of whether the scheme is "better" (faster, easier-to-compute, smaller key sizes, smaller ciphertext sizes, or something along these lines).

If the proposed scheme doesn't rely on one of these hard problems, then it will take years and years for people to really look twice at it. This is because historically, most proposed hard problems haven't actually been so hard after all. The current lattice-based cryptosystems were first proposed in a modern sense with NTRU in the early 90s, and it wasn't until over 10 years later that people began to think that lattice-based cryptography might work out.

To generate improved implementations of a current cryptosystem is an important task, but more often this falls to computer scientists and cryptographers than raw number theorists. Equally important are implementation-based attacks on cryptosystems (as opposed to breaking the underlying mathematics) --- a true implementation of RSA will include padding and time obfuscation to obscure timing attacks. This is also frequently the domain of computer scientists and cryptographers.

To generate keys for a given cryptosystem is usually very easy. One might read the specification for the cryptosystem and follow it. Deviating from the spec is usually an insecure idea. But there are classes of keys which are advised against for little reason other than some heuristic, and it is conceivable that one might find it fun to improve understanding of key generation. But typically people find it both more fun and more approachable to prove sets of keys as bad, since this usually amounts to coming up with some clever attack.

Finally, very many mathematicians, computer scientists, programmers, and cryptographers think about these topics all the time. They have conferences, journals, their own jargon, and their own habits. If one hopes to add their collective body of knowledge, it is necessary to engage with them through the same major channels: conferences, journals, etc. You should expect that the difficulty and challenges facing an amateur cryptographer to be roughly analogous to the difficulty and challenges facing an amateur mathematician. These challenges are not insurmountable, but also not small.