Differentiation $\frac {d}{dx}$ and integration $\int$ in standard calculus are "opposite operations" in some sense, but not literally inverses (because of the constant of integration).
This makes me guess that there is a sense in which they are adjoint (in the sense of category theory). Is there a way to formalize this?
Yes, there is a sense in which they are adjoint, and in fact they form an equivalence.
In fact there is a more general construction : if $X,Y$ are sets and $R,S$ are equivalence relations on $X$ and $Y$ respectively, then $R$ can be seen as a thin category whose set of objects is $X$ and there is an arrow $x\to x'$ if and only if $xRx'$ (note that the symmetry of $R$ implies that it is in fact a thin groupoid). If we view $S$ as a category in the same way, then a functor $R\to S$ is just a function $\alpha:X\to Y$ such that $xRx'$ implies $\alpha(x)S\alpha(x')$ for all $x,x'$. In particular, an equivalence between the categories $R$ and $S$ is the same thing as two functions $\alpha:X\to Y$ and $\beta:Y\to X$ satisfying the property above, and such that $xR\beta(\alpha(x))$ and $yS\alpha(\beta(y))$ for all $x\in X$ and $y\in Y$.
Now to get your example, define $X=C^1(\mathbb{R,R})$, $Y=C^0(\mathbb{R,R})$, $R$ is the equivalence relation $$fRg\Leftrightarrow f-g \text{ is constant,}$$ $S$ is just the discrete relation on $S$ (i.e. equality), $\alpha$ is the differentiation, and $\beta$ is "the" integration (i.e. $\beta$ just picks some primitive for every function).