I'm trying to learn basic logic, and it seems that Peano Arithmetic — a first order theory of the natural integers with $+$ and $\times$ symbols, is a common object of interest.
We could define (a priori) richer theories of arithmetic, for instance if we introduce the operation $x^y$ (exponentiation).
Also, we could add axioms for negative numbers and the $-$ symbol and define a theory of which $(\mathbb{Z}, +, -, \times)$ would be a model, for instance.
Wouldn't those two theories have theorems that are not theorems in Peano Arithmetic? If so, why is only Peano Arithmetic presented in all logic courses?
Usually by PA, we understand that we are including the induction axiom in the theory, as in the wiki page. If this is the case, you could define $+, \times$, exponential, all by induction, as can be see in the wiki. So adding exponential doesn't make it stronger.
A common motif is not use the induction axiom, as it is too strong. In this case we could work with the Robinson arithmetic (theory Q), which is as incomplete as PA. As commented, we can define exponentiation here also by the $\beta$-function.
But, because Peano arithmetic (or Q) is incomplete, there is a Gödel sentence there $G$ that the system can't prove or refute. So we could add $G$ to PA as an axiom. This system would be stronger than PA, because PA + G would proof G and PA alone could not proof it. But PA + G would also be incomplete, so we could add G', and so on. In this way we can get a infinite sequence of stronger theories.
We are still in the realm of same language theories. But in fact theories with different languages, as one with the $-$ sign could be compared with arithmetic, if we can define a sufficient amount of arithmetic there. The same happens with the ZFC axioms for set theory, which is incomplete. We can create symbols in ZFC that would act exactly as the ones in PA. Specifically: the set of finite ordinal numbers {0,1,2,...} (denoted $\omega$) can be defined in ZFC, and also the + and $\cdot$ operations. In ZFC you can prove that all of PA holds for the structure $(\omega,+,\cdot)$. We say that we've interpreted PA in ZFC.
In ZFC you can prove many things about PA (thus interpreted) that cannot be proven in PA. But this "PA in ZFC" is still incomplete: there are sentences of PA that cannot be decided one way or the other using ZFC.
So PA (or Q) is important because it can show that another systems that have it as subsystems are incomplete. And of course there are other interests besides incompleteness results.