My favorite proof that $e$ is irrational goes something like this. Observe that we can write any real number $r$ as $$ a \,+\, \frac{b_2}{2} \,+\, \frac{b_3}{3!} \,+\, \frac{b_4}{4!} \,+\, \frac{b_5}{5!} \,+\, \cdots $$ where $a\in\mathbb{Z}$ and each $b_n\in\{0,1,\ldots,n-1\}$. This is the expansion of $r$ in the factorial number system, where $b_n$ is the $n$'th "digit". In particular, $$ a \;=\; \lfloor r\rfloor\qquad\text{and}\qquad b_n = \big\lfloor n!\,(r-s_{n-1})\big\rfloor $$ for each $n$, where $s_n$ denotes the $n$'th partial sum of the above series.
It is easy to see that $r$ is rational if and only if this expansion terminates. Then $e$ must be irrational, since its expansion does not terminate: $$ e \;=\; 2 \,+\, \frac{1}{2} \,+\, \frac{1}{3!} \,+\, \frac{1}{4!} \,+\, \frac{1}{5!} \,+\, \cdots $$ Question: Can this proof somehow be modified to show that $e^2$ is irrational? It doesn't work straight off, since $2^n$ isn't in the range $\{0,1,\ldots,n-1\}$.
I'm also curious whether there are any other cases in which irrationality of an interesting number can be proven using a non-standard positional numeral system.
Not quite $e^2$, but consider $x = a e + b/e$ for integers $a,b$ with $|b| \le a$. Then $x = \sum_{j=0}^\infty c_j/j!$ where $c_j = a+b$ when $j$ is even and $a-b$ when $j$ is odd. Write $x = r + y$ where $r = \sum_{j = 0}^{a+b} c_j/j!$ is rational and $y = \sum_{j=a+b+1}^\infty c_j/j!$ is a non-terminating expansion in factorial base, and thus is irrational. Therefore $x$ is irrational.