In the standard abstract algebra curriculum, one learns a battery of irreducibility tests for factoring polynomials over $\mathbb{Z}$ (equivalently, by Gauss' lemma, over $\mathbb{Q}$). For instance (not all names standard):
- Linear Factor Test: A polynomial has a linear factor over $\mathbb{Z}$ if and only if it has a root in $\mathbb{Q}$.
- Quadratic/Cubic Test: A polynomial of degree 2 or 3 is reducible if and only if it has a linear factor.
- Brute Force Method: Write out the forms of all possible factorizations. For instance, after checking a quartic for linear factors, look at $(X^2+aX+b)(X^2+cX+d)$. Obtain a system of equations for the coefficients. Determine whether solutions exist. Ugh. (Though, more feasible over $\mathbb{Z}_p$.)
- Mod-$p$ Irreducibility Test: If there exists a prime $p$ such that a polynomial is irreducible over $\mathbb{Z}_p$, then it is irreducible over $\mathbb{Z}$.
- Eisenstein's Criterion: If there exists a prime $p$ which divides all but the lead coefficient, and whose square does not divide the constant term, then the polynomial is irreducible.
- Substitution tricks: The reducibility of a given polynomial $f(X)$ is related to the reducibility of other polynomials like $f(aX+b)$ or the reversal $X^n f(1/X)$.
- Complexify: Factor the polynomial into linear factors over $\mathbb{C}$. Every higher-degree divisor of the polynomial is a product of several of these linear factors. Try out all the products of the linear factors and verify that all of them have non-integer coefficients. (See for instance Jyrki Lahtonen's solution in this post).
- Special cases: E.g. cyclotomic polynomials are something you should just know.
These are often used in combination. One can prove that $X^4+X+1$ is irreducible over $\mathbb{Z}$ by showing it's irreducible over $\mathbb{Z}_2$, which in turn can be easily done by the "brute-force" approach since there are very few quadratics over $\mathbb{Z}_2$. Jyrki Lahtonen's solution in this post shows that $f(X):=X^4-10X^2+1$ is irreducible by applying Eisenstein with $p=2$ to the reversal of $\frac{1}{8} f(2X+1)$. (Gorgeous!)
How does one get a sense for which tricks to try when? There are infinitely many primes $p$ to try with Eisenstein and mod-$p$ tests, though in practice $p$ tends to be small. Allowing substitution tricks opens up a dizzying array of possibilities. I'm thinking by analogy with the convergence tests one learns in calculus. One can just try different approaches until one of them works, but one can also see patterns: a series with powers or factorials is likely amenable to the Ratio Test, terms of "smaller order" can be eliminated by the Limit Comparison Test, terms that you know how to bound can be handled by the Comparison Test, and functions you know how to integrate are promising candidates for the Integral Test.
Are there analogous clues to look for here? How might you "smell" which test is likely to work with which polynomial?
None of these methods work in general. You can write down irreducible polynomials, starting I think at degree $4$, which are reducible $\bmod n$ for every positive integer $n \ge 2$, and hence on which both Eisenstein's criterion and reducing $\bmod p$ necessarily fail. There is some theory of Newton polygons over the $p$-adics which is supposed to generalize both of these but I never learned it.
Checking for irreducibility is quite hard in general, and honestly I still don't have great intuitions about what method to use when, I just throw everything I know at a polynomial until something sticks. Here's a great example: on MO someone asked whether or not the polynomial
$$x^n + p_1 x^{n-1} + p_2 x^{n-2} + \dots + p_n$$
is always irreducible, where $p_i$ is the $i^{th}$ prime. I observed in the comments that the constant coefficient being prime means that, if the polynomial were reducible, its irreducible factors must have constant term $\pm 1$ or $\pm p_n$, and the latter case occurs exactly once; in particular you can deduce that at least one complex root must be inside (or on) the unit circle and at least one complex root must be outside (or on) the unit circle. So if it were possible to rule that out, the polynomial is irreducible. And then Bjorn Poonen was able to rule this out, by showing that all of the roots of this polynomial lie outside the unit circle.
The reason it occurred to me to think about the location of the complex roots is by an analogy to the proof of Perron's criterion, which says the following.
This can be proven using Rouche's theorem to show that the above condition implies that exactly one of the roots of $P$ lies outside the unit circle, and the rest must lie strictly inside. It doesn't come up all that often in practice, though.