I'm curious about $$ \lim_{n \rightarrow \infty} \mathbb{F}_n $$ Is it $\mathbb{Z}$? That seems reasonable if you consider it as a set but of course $\mathbb{Z}$ is not a field so that is confusing. I think the problem is probably how you define the limit in this case. Has anyone ever done so?
Edit: Another question. What is the smallest field that contains all finite fields? We think the answer to this is $\mathbb{Q}$, but again we don't have a formal definition of "containment", so this is a problem too. Maybe using subfields. Have either of my questions ever been studied?
There are many reasonable notions of "limit of fields" depending on your application. Often, "limits" of sets/spaces/algebras are best thought of in terms of unions. For example, fix a prime $p$, and consider the fields $\{\mathbb F_{p^n}\}_{n\in \mathbb N}$ of order $p^n$. If my memory is correct, there is a map $\mathbb F_{p^n} \to \mathbb F_{p^m}$ if and only if $n|m$. Such a map is necessarily an inclusion, but it is not quite unique: you could first apply some field automorphism of $\mathbb F_{p^n}$. But you can build the finite fields in such a way so as to have a canonical such map. Then this collection is together a "directed set": it is a partially ordered set, and any two points therein have an upper bound. With these extra inclusions, you can form the "union" of all of the $\mathbb F_{p^n}$s as follows: you take the set-theoretic disjoint union, but then identify $x\in \mathbb F_{p^n}$ with $y\in \mathbb F_{p^m}$ if $x$ and $y$ become equal upon embedding both fields into some larger $\mathbb F_{p^\ell}$. The result of all of this is that you get the algebraic closure $\overline{\mathbb F_p}$.
When you want to consider different primes at the same time, this approach doesn't work, and in any case it might not have been the answer you wanted for your application. Here's a different option. Let me abbriviate by $\mathbb P = \{2,3,4,5,7,8,9,11,13,16,\dots\}$ the set of prime powers. It is a countable set. For any set $X$, I will write $\mathcal P(X)$ for the set of subsets of $X$. Define a subset $U \subseteq \mathcal P(\mathbb P)$ to be a proper filter if it satisfies: (i) $\emptyset \not \in U$. (ii) If $A,B \in U$, then $A\cap B \in U$. (iii) If $A \in U$ and $B \supseteq A$, then $B \in U$. In particular, there is a filter called $\mathrm{Cof}$ defined by $$ \mathrm{Cof} = \{A \in \mathcal P(\mathbb P) : |\mathbb P \smallsetminus A|<\infty\} $$
Filters should be thought of in terms of voting. The population of your country is the set $\mathbb P$. The elements of $U$ are the "winning coalitions" for some democratic voting system. A subset of $\mathbb P$ is a "proposition", or rather it is the subset of voters who support a given proposition. Condition (i) says that if no one votes for a given proposition, then that proposition fails. Condition (iii) says that if you increase the support for a proposition that was already winning, then it remains winning. Condition (ii) is the strongest: it says that if proposition A and proposition B both pass when voted on independently, then the combined proposition (A and B) also passes.
Finally, define a filter $U$ to be ultra if it satisfies (iv) if $A \not \in U$, then $\mathbb P \smallsetminus A \in U$. This says that if a proposition does not pass, than the opposite proposition would. This rules out that government would get stuck like the current US Senate. Arrow's theorem says that if the population $\mathbb P$ were finite, then the only ultrafilters would be dictatorships — the winning coalitions are the ones that include the dictator, and no one else's vote counts. But since $\mathbb P$ is infinite, there do exist non-dictatorial ultrafilters. Indeed, the Axiom of Choice implies that any filter can be extended to an ultrafilter (because the union of an ascending chain of filters is a filter), and any ultrafilter containing $\mathrm{Cof}$ is not dictatorial. (Indeed, ultrafilters on $\mathbb P$ are either dictatorial or contain $\mathrm{Cof}$.)
Ok, so let's use the axiom of choice to build a non-dictatorial ultrafilter $U$. I will use it to build a field which is a "limit" of the finite fields. Here's how. Consider first the set $\prod_{q\in \mathbb P} \mathbb F_q$ of all $\mathbb P$-indexed sequences $\{x_q \in \mathbb F_q\}_{q\in \mathbb P}$. I.e. $x_2 \in \mathbb F_2$ and $x_3 \in \mathbb F_3$ and so on. This set is a ring: $(x+y)_q = x_q + y_q$ and $(xy)_q = x_qy_q$. It is not a field: $(0,1,0,0,0,\dots)$ is non-zero but not invertible (since it is a zero-divisor; for example, $(1,0,0,0,\dots) (0,1,0,0,0,\dots) = (0,0,0,0,\dots)$.
To make it a field, we should quotient out by a maximal ideal. I will define an ideal $I_U \subseteq \prod_{q\in \mathbb P} \mathbb F_q$ so that $$ x \in I_U \Leftrightarrow \{q \in \mathbb P : x_q = 0\} \in U.$$ What I'm saying is that every voter in $\mathbb P$ votes on the question of whether $x$ "looks like $0$", and if a majority votes "yea", then that element is in $I_U$. Conditions (ii) and (iii) imply that $I_U$ is an ideal. Condition (i) implies that $I_U$ is a proper ideal. Condition (iv) implies that $I_U$ is a maximal ideal. You should check these as exercises if they are not obvious.
The quotient $(\prod_{q\in \mathbb P} \mathbb F_q) / I_U$ is called the ultraproduct of $\{\mathbb F_q\}$ along $U$. It is a field, and behaves in many ways like a "limit" of the $\mathbb F_q$s. For example, suppose that some algebraic fact is true for all $\mathbb F_q$ past some point --- for example, whether some polynomial has a solution. Then that fact holds for for the ultraproduct. Indeed, for any "finite" statement (i.e. expressible in the first-order language of fields), it is true in the ultraproduct iff the proposition "that statement is true" passes when all voters vote upon it.
Now, depending on what ultrafilter you use, the ultraproduct will be very different. For example, you can find an ultrafilter so that only the voters who are powers of $5$ count. Then the resulting ultraproduct will be a field of characteristic $5$, because a winning coalition of voters believe it to have characteristic $5$. You can find an ultrafilter for which only the voters of the form $5^{2^k}$ count. Then the only polynomials irreducible over over $\mathbb F_5$ that split over the ultraproduct will be towers of quadratics --- the Galois group will be a 2-group. On the other hand, you can find ultrafilters for which the winning coalitions tend to have voters from all demographics (i.e. all primes, etc.). Then the ultraproduct will be a field of characteristic $0$. You can even arrange for the ultraproduct to be algebraically closed, in which case it will be (highly non-canonically) isomorphic to $\mathbb C$.