My (maybe wrong) impression is that while probability is widely used in science (for example, in statistical mechanics), it is rarely seen in pure mathematics. Which leads me to the question -
Are there some interesting application of Probability Theory in pure mathematics, outside Probability Theory itself?
On
One of my favorite examples is a proof of the Weierstrass approximation theorem, the fact that polynomials are dense in the space of continuous functions over an interval.
I like the proof because it's elementary and the conclusion comes as a surprise.
On
You may find some interesting examples on this MathOverFlow thread:
https://mathoverflow.net/questions/9218/probabilistic-proofs-of-analytic-facts
On
There is a strong connection between probability theory and areas of real analysis such as linear PDE and potential theory. If $X_t$ is a continuous time stochastic process on a state space $S$, the function $u(x,t) = E_x[f(X_t)]$ tends to satisfy an equation like $\partial_t u + Lu = 0$, where $L$ is some interesting operator. ($E_x$ denotes that the process should start at the point $x \in S$, and $f$ is a real-valued function on $S$.) The classical example is when $X_t$ is Brownian motion on $\mathbb{R}^n$; then $L$ is the Laplace operator. So one can study heat equations, harmonic functions, spectral theory, functional inequalities, and lots of other "analysis" topics by thinking about a corresponding stochastic process.
The theory of Dirichlet forms makes this correspondence quite precise.
If the state space $S$ is a manifold, then the study of the operator $L$ tends to be closely related to the geometry of $S$, so one can also study differential geometry through probability theory.
On
You might find interesting various probabilistic plausibility arguments for the $\rm\:3n+1$ conjecture. For an introduction see Lagarias's survey excerpted below, and for more sophisticated arguments see his paper How random are the 3x+1 function iterates?.
1. Introduction.$\;$ The $\rm\:3x + 1\:$ problem, also known as the Collatz problem, the Syracuse problem, Kakutani's problem, Hasse's algorithm, and Ulam's problem, concerns the behavior of the iterates of the function which takes odd integers $\rm\:n\:$ to $\rm\:3n + 1\:$ and even integers $\rm\:n\:$ to $\rm\:n/2\:$. The $\rm\: 3x + 1\:$ Conjecture asserts that, starting from any positive integer $\rm\:n\:$, repeated iteration of this function eventually produces the value $\rm\:1\:$.
The $\rm\:3x + 1\:$ Conjecture is simple to state and apparently intractably hard to solve. It shares these properties with other iteration problems, for example that of aliquot sequences (see Guy [36], Problem B6) and with celebrated Diophantine equations such as Fermat's last theorem. Paul Erdos commented concerning the intractability of the $\rm\:3x + 1\:$ problem: "Mathematics is not yet ready for such problems." Despite this doleful pronouncement, study of the $\rm\:3x + 1\:$ problem has not been without reward. It has interesting connections with the Diophantine approximation of $\log_2 3\:$ and the distribution (mod $\rm\:1\:$) of the sequence $\rm\:\{(3/2)^k : k = 1,2,\ldots\}\:$, with questions of ergodic theory on the $\rm\:2\:$-adic integers $\rm\:\mathbb Z_2\:$, and with computability theory--a generalization of the $\rm\:3x + 1\:$ problem has been shown to be a computationally unsolvable problem. In this paper I describe the history of the $\rm\:3x + 1\:$ problem and survey all the literature I am aware of about this problem and its generalizations.
2.1. A heuristic argument. $\;$ The following heuristic probabilistic argument supports the $\rm\:3x + 1\:$ Conjecture (see [28]). Pick an odd integer $\rm\:n_0\:$ at random and iterate the function $\rm\:T(n) = n/2\ \text{if n is even}\:$ $\rm\:\text{else}\ (3n+1)/2\:$ until another odd integer $\rm\:n_1\:$ occurs. Then $\rm\:1/2\:$ of the time $\rm\:n_1 = (3 n_o + 1)/2,\ 1/4\:$ of the time, $\rm\:n_1 = (3 n_o + 1)/4,\ 1/8\:$ of the time $\rm\:n_1 = (3 n_o + 1)/8\:$ and so on. If one supposes that the function $\rm\:T\:$ is sufficiently "mixing" that successive odd integers in the trajectory of $\rm\:n\:$ behave as though they were drawn at random (mod $\rm\:2^k\:$) from the set of odd integers (mod $\rm\:2^k\:$) for all $\rm\:k\:$, then the expected growth in size between two consecutive odd integers in such a trajectory is the multiplicative factor
$$\frac{3}{2}^{1/2}\frac{3}{4}^{1/2}\frac{3}{8}^{1/2}\cdots \ = \ \frac{3}{4} < 1\:.$$
Consequently this heuristic argument suggests that on average the iterates in a trajectory tend to shrink in size, so that divergent trajectories should not exist. Furthermore it suggests that the total stopping time $\rm\:\sigma_x(n)\:$ is (in some average sense) a constant multiple of $\rm\:\log n\:$.
From the viewpoint of this heuristic argument, the central difficulty of the $\rm\:3x + 1\:$ problem lies in understanding in detail the "mixing" properties of iterates of the function $\rm\:T(n)\ (mod\ 2^k)\:$ for all powers of $\rm\:2\:$. The function $\rm\:T(n)\:$ does indeed have some "mixing" properties given by Theorems B and K below; these are much weaker than what one needs to settle the $\rm\:3x + 1\:$ Conjecture.
On
In addition to examples, the overall "gestalt" answer to the question is: probability is used everywhere in mathematics. It is a basic idea on the level of algorithm, algebraic structure, geometry, calculus, or other very ubiquitous things. It has become a very popular source of questions and intuitions in research, in all fields. Knowing that this is true, it is not surprising that many examples of theoretical uses of probability can be posted.
Being a basic language, it is also true that many of the uses of probability are basic, and do not go far beyond the idea of a probability distribution, frequencies of events, expectations, related combinatorics and so on. But in some fields, advanced results in probability theory are constantly being used.
On
The Erdős–Kac theorem shows that the (log of the log of the) prime factors of a number are Poisson/normally distributed.
The topic of Probabilistic Combinatorics. See http://en.wikipedia.org/wiki/Probabilistic_method
This is a powerful way of giving non-constuctive existence proofs for lots of different (finite) mathematical structures and determining their properties.
I found this the highlight of all my undergraduate probability courses. If you are interested in learning more I would recommend: "The Probabilistic Method" by Alon and Spencer.