Is arithmetic special for logic?

Question

I'm beginning learning logic, some theorems such as Goedel's two incompleteness theorems are stated as being "about" arithmetic. However as I understand things, other first order theories could give analogous results, some of those theories are much simpler like this one of concatenation. Arithmetic is also used for examples such as what we can do with the completeness theorem. Is this purely because arithmetic was historically important or is there a deeper connection between arithmetic and logic?