I have encountered a number of situations where the solution of a PDE and a certain expectation associated to a Markov process are equal. Two examples include:
The heat equation $u_t = \frac{1}{2} \Delta u$ with initial data $u(0,x)=f(x)$, considered on the whole space. Here the solution is given by $u(t,x)=\mathbb{E}(f(x+W_t))$ where $W_t$ is a Wiener process.
Suppose $U$ is a bounded open set, $X_t^x$ is a Markov process which starts at $x$ and has generator $L$, and $\tau_x = \inf \{ t \geq 0 : X_t^x \in \partial U \}$. Then $u(x) = \mathbb{E}(\tau_x)$ solves
$$(Lu)(y) = -1 \text{ if } y \in U \\ u(y) = 0 \text{ if } y \in \partial U$$
for $x \in \overline{U}$. When $X_t^x$ is a diffusion process this is an elliptic or semielliptic boundary value problem.
The most general formula for such situations that I have seen is the Feynman-Kac formula, though I have seen hints that the Feynman-Kac formula is itself a special case of Girsanov's formula, which I have some difficulty understanding. I am looking for a relatively comprehensive discussion of this phenomenon, preferably with some applications or at least additional toy examples.
For future reference, I found a nice source myself:
http://www3.ntu.edu.sg/home/nprivault/papers/greifswald_potential.pdf
Sections 1 and 2 are on potential theory. Sections 3 and 4 are on probability. Section 5 presents a number of the sorts of connections I was looking for here. A bit of overlap is splashed between the first four sections. It is a very nice source although it would have been better for my purposes if Section 5 had a bit more content.