I think if $A$ is the usual generator for the Markov process $(X_t)_t$
$$A f (x) = \lim_{t \downarrow 0} \frac{\mathbb{E}^{x} [f(X_{t})] - f(x)}{t}$$
then we get that for any "nice" $f$ the process
$$f(X_t)-\int_0^t Af(X_s)ds$$
is a continuous local martingale. This makes intuitive sense, but what theorems can I apply to see that it's true? FTC was mentioned but I don't know how to apply that to expressions like $dt \cdot Af(X_t)$
Set $$M_t := f(X_t) - \int_0^t Af(X_s) \, ds$$ and write $P_t f(x) := \mathbb{E}^x f(X_t)$ for the semigroup associated with $(X_t)_{t \geq 0}$. Using the definition of the generator, it is not difficult to see that
$$\frac{d}{dt} (\mathbb{E}^x f(X_t)) = \frac{d}{dt} P_t f(x) = P_t Af(x).$$
By the fundamental theorem of calculus, this implies
$$P_t f(x)-P_s f(x) = \int_s^t \frac{d}{dr} P_r f(x) \, dr = \int_s^t P_r Af(x) \, dr. \tag{1}$$
Now we are ready to show that $(M_t)_{t \geq 0}$ is a martingale (if $f$ is nice). By the Markov property, we have
$$\begin{align*} \mathbb{E}^x(M_t \mid \mathcal{F}_s) &= \mathbb{E}^{X_s} f(X_{t-s}) - \int_0^s Af(X_r) \, dr - \mathbb{E}^x \left( \int_s^t Af(X_r) \, dr \mid \mathcal{F}_s\right) \\ &= P_{t-s} f(X_s) - \int_0^s Af(X_r) \, dr - \int_0^{t-s} \mathbb{E}^x(Af(X_{s+r}) \mid \mathcal{F}_s) \, dr \\ &= P_{t-s} f(X_s) - \int_0^s Af(X_r) \, dr - \int_0^{t-s} \underbrace{\mathbb{E}^{X_s}(Af(X_r))}_{P_r Af(X_s)} \, dr \\ &\stackrel{(1)}{=} P_0 f(X_s) - \int_0^s Af(X_r) \, dr = M_s. \end{align*}$$
For more details see e.g. René Schilling, Lothar Partzsch: Brownian Motion - An Introduction to Stochastic Processes, Chapter 7.