I've been studying 1's and 2's complement, but they still seem mysterious to me even though I completely understand how to manipulate them.
They seem like cute tricks that somehow just "work out".
I was wondering if there was a group theoretic / algebraic way of looking at the complementing operations that would explain the "why" of their working
I was hoping for a formulation that answers the questions of
Why does 2's complement preserve uniqueness of numbers?
Why is it that negative numbers "just work out" in 2's and 1's complement?
Can these be generalized to n's complement?
Are there interesting algebraic structures created from this number representation system?
Why are there negative integers? Because there are infinitely many integers. $\mathbb Z$ is an (the) infinite cyclic group. You have "negative" and "positive" because the integers extend infinitely to each side of zero (in spite of the name "cyclic"). What happens in finite groups? There is no clearly defined positive or negative, because if you keep going in one direction, you keep passing zero again and again. So in $\mathbb Z_n$, $-1$ is $n - 1$; $-2$ is $n - 2$, and so on. This is exactly what is used in one's complement. For an $n$-bit one's complement representation, take $\mathbb Z_{2^n - 1}$. Then, $-1$ is $2^n - 2$, which in binary, is precisely the bitwise complement of $1$ with respect to $2^n - 1$. And similarly for other "negative" numbers. Addition is obviously modulo $2^n - 1$, which is why there is a "cyclic carrying" (from the MSB to the LSB) in case of overflow.
Example: In $3$-bit one's complement representation, we use \begin{equation*} \mathbb Z_7 = \{0, 1, 2, 3, 4, 5, 6, 7=0\} = \{000, 001, 010, 011, 100, 101, 110, 111=000\}. \end{equation*} We are forced to include the superfluous $111$, because we have chosen a modulo $7$ rather than modulo $8$ representation (or in general, modulo $2^n - 1$ rather than modulo $2^n$). Of course, we could choose to use any other finite cyclic group without any theoretical issues. But then negation will no longer be bitwise complementation (recall that it is so here only because we are subtracting from $2^n - 1 = \underbrace{11\ldots1}_{n}$).
But why not take $\mathbb Z_{2^n}$, in order to avoid the ambiguous representation of zero? Why not indeed, then you get two's complements!
I believe I answered your first, second, and even the fourth question, since the algebraic structure is just a finite cyclic group. As for generalization to $n$'s complement, that would make sense (or be useful) in a base-$n$ system. In binary, you have one's complement (complement from $2^n - 1$, which is equivalent to bitwise complement from $1$) and two's complement (complement from $2^n$, which is bitwise complement from $2$). In the decimal system, you can similarly have ten's complement (note that even in binary, "two's complement" is actually "10's complement"), nine's complement, etc. For example, using $3$-digit nine's complement subtraction, \begin{equation*} 251 - 123 = 251 + 876 =(1)127 = 128 \end{equation*} (where the $(1)$ is the cyclic carry).